---
title: "How to Validate AI-Enabled, Non-Product Software, Fast"
type: webinar-transcript
publisher: Ketryx
source: "https://fast.wistia.net/embed/iframe/xcji2qni5a"
content: auto-caption transcript, proper-noun corrected
---

# How to Validate AI-Enabled, Non-Product Software, Fast

*Ketryx webinar — transcript of the recorded session.*

[▶ Watch the recording](https://fast.wistia.net/embed/iframe/xcji2qni5a)

---

Well, you very much for joining us here today. We're really excited to get into some of the content and hope you are as well. Today's session is focused on how to validate AI enabled product software, the internal AI tools, agents, and co pilots increasingly running inside your regulated workflows. I'm Richard Schmidt, Director of Delivery and Success here at Ketryx, and I work closely with quality and regulatory teams navigating compliance for AI enabled tools. We'll walk through why validation breaks for nondeterministic systems, a risk based approach you can apply today, and what it looks like to put this practice in place with real customers.

James and I will get started in just a few minutes. And maybe as we get started, James, you work with a lot of regulated teams day to day. Does the FDA actually expect full IQ, OQ, PQ for every AI tool teams use internally? Hey, guys. And, yeah, Rich, it's a good question, and the answer is gonna be no.

So the FDA's CSA computer software assurance guidance was actually designed to kill reflective IQ OQPQ for everything. The expectation is now to match your assurance to the risk of the intended use. So there are three tiers in this guidance. First is gonna be that low risk where the AI outputs are advisory. A human's gonna verify before anything actually enters a quality record.

So things like drafting a risk analysis with Claude and then an engineer reviewing and approving it. So your SOP and the review step are actually all the assurance. There's actually no IQOQPQ needed. For medium risk items, that's things like AI feeding a control process with reduced oversight, such as auto classifying complaints. For these, the guidance is to do focus testing with a representative dataset, doc acceptance criteria, and just monitor periodically.

And then there are those high risk applications. This is for when AI is influencing patient safety decisions. For this, the guidance is to do full rigor, but CSA still says critical thinking and risk based decisions take precedence over just checkbox docs. The mistake I think that most companies make is treating every AI tool like it's high risk. That's not what the FDA wants.

All that does is generate documentation that nobody reads instead of genuine assurance. So the question shouldn't be, did we do IQ, OQ, PQ? It's more so, do we have the objective evidence that this tool is fit for its intended use and is proportional to the risk? Great, thank you. Well, as I mentioned, my name is Richard, so maybe just a quick bit of background on me.

As you can see, I was fortunate to work with Veeva in their life sciences and R and D, and I spent some time in quality, regulatory, clinical operations, safety and was eventually a Vault architect. From there I moved further upstream into clinical trial design at a company called Faroe Health and there I was leading customer strategy as we were modernizing the infrastructure and data driven protocols were starting to gain traction. I was lucky enough to work with teams who are optimizing their clinical development tools and processes. And now I'm really excited to be contributing to Ketryx's journey, working in delivery and success. As you know, we work with four of the top five medical device manufacturers and customers who are using modern tech to accelerate regulated product development and streamline cloud operations.

In Delivery and Success, I'm responsible for making sure that the platform is deployed and our customers are happy. And, James, I'll hand over to Yeah. Hey, everyone. James, excited to be here. Quick background on me.

I spent started my career over at PwC on the SAP implementations team, helping large enterprises navigate complex technology transformations. From there, I moved over to Ketryx first as a forward deployed engineer on the delivery of success team, working hands on with some of the largest medical device manufacturers in the world to help accelerate their product development while staying compliant. And now as a solutions engineer at Ketryx, I get to work with teams directly on figuring out how to bring AI into their regulated workflows, which is exactly what we we'll be talking about today. Thanks, guys. Excited to be here.

Great. Thank you. Now, let me introduce Ketryx is an AI native compliance platform purpose built for regulated teams to deliver safer products faster. We interoperate with your tooling across various teams throughout your organization and apply your quality rules across each team's critical toolset. With that data, we can automatically generate the compliance evidence and documentation you need to stay audit ready.

And with our AI, we can automatically trace requirements, assess change impacts, validate test coverage, and flag compliance risks in real time. Ketryx itself is certified to IEC sixty two thousand three hundred and four, ISO thirteen thousand four hundred eighty five, ISO seventeen thousand nine hundred seventy one and twenty seven thousand and one. And what this means for your quality teams, there's less manual chasing of evidence and more confidence in audit or submission readiness. Well, let's start with first defining what non product software is. Non product software is anything in your GXP environment that doesn't ship with the device.

So that could include your QMS, LIMS, manufacturing systems, Jira for project management, but it also includes the AI code someone on your team wrote three months ago to parse batch data and the internal chatbot that answers your SOP questions. Here's the test we use. If it influences a GXP decision and it's not part of the product, it's non product software. Under medical device standards, it requires validation. The reason this matters right now is that the volume of non product software in your environment is growing very quickly.

Teams used to maintain validation for two or three internal systems, and that was the job. Now there's an LLM powered tool embedded in every team's workflow, whether that's engineering, quality, regulatory, or manufacturing. The scope of what counts as non product software didn't expand, but the number of tools you're using that fall inside that scope absolutely did. Most importantly, product software includes the things your team built internally. A Python script that analyzes batch data, an internal chatbot that answers your quality questions, a validation assistant built on top of an LLM.

If it's influencing a GXP decision and it's not part of the product, it's non product software and it requires validation. So why modernize validation right now? Validation isn't free. Gartner says it adds up to forty percent to project cost. McKinsey says quality assurance work consumes more than twelve percent of industry revenue.

Now let's look where the spend is going. The FDA's CSA pilot data shows that eighty percent of validation test cases are low risk, which means they can be automated and scripted without losing any rigor. Low risk software is consuming the time and budget that your higher risk systems actually need. That's the structural problem in this industry today. Teams are working hard.

They're just working **** ** the wrong distribution of effort. This isn't a gradual shift and the FDA is now actively encouraging the move. Your team is going to need to move to a risk based approach and the details really do matter. We'll get into how teams can apply a risk based approach in just a moment. But before we do, let's take a quick look at the agenda for today.

What we'll be going over. We'll start with the challenges, which is why validating AI enabled non product software is so difficult and why adoption in regulated environments is harder than it looks. From there, we'll talk about the common gaps that teams run into when they're validating non deterministic software. We'll also get into how to take a risk based approach to validation, looking at what a compliant framework actually looks like in practice without creating unnecessary overhead. And we'll close with proven strategies for validating non product software at scale so you can deploy these systems safely and efficiently across your organization.

Before we get into the content, we'd love to hear from you. What are your biggest challenges validating nonproduct software today? It could be AI enabled or it could be more traditional. But given the way the industry is moving, I'd guess most of you are asking this question about AI right now. So we'll let this poll run for a moment.

And while it's running, James, you talk to regulated teams about this every week. What's the version of this conversation you hear most? Yeah. It's a good question, Rechio. I think that I hear the same version, excuse me, over and over again.

Customers want to use AI. They're also being pushed to use AI because they think it's gonna help them develop faster. The thing is that most teams can't keep up with the constant pace of software change. That's part of why so many teams are still on Jira data center instead of Jira cloud. Teams used to wait for Jira updates on a quarterly cadence or maybe twice a year, but now every time you log in to Claude, it feels like there's a new version.

Sometimes it happens literally every twenty minutes. So the questions that we're hearing are, what happens when Anthropic releases a new update and the model starts to drift or hallucinate? How do you keep that validation current when the soft software is updating beneath you? And the question that that they can't seem to answer that they're all kind of pointing to is, how do you actually validate it, and how do you actually test it? Great, thank you.

What are your biggest challenges validating non product software today? Risk assessment and scoping, what needs validation? Test planning and execution effort? Documentation and traceability burden, defining acceptance criteria for AIML outputs, and monitoring AIML models in production. We've got a good distribution right now leading at thirty nine percent is what I see right now is the risk assessment and scoping what needs validation.

That's very interesting. Actually, I think we will cover off on that very well today in the session. Thank you all for your participation. That's very telling. Appreciate the engagement there.

So let's go ahead and continue. Why is validating AI in a GxP environment so difficult? Validating non product software was already hard before AI came along, and AI just happened to turn up the rate of change on every part of it. Traditional computer systems validation or CSV was built for a deterministic world. You write user requirements, your functional specs, your code, and then you test that the system does exactly what you told it to every single time, same input, same output, and that's the V model.

You can see the V model over here on the right. AI breaks that in five different places. The first is where it breaks it in change management. Change management is now frequent. Traditional software releases on a schedule, but AI updates as new data comes in on a cadence that you don't set.

The second is that external models update silently. As you heard from James, if you're building on Gemini or Claude or ChatGPT, those models are changing in the background outside of your control, and you don't get a release note. The third is the perception problem. There is a widespread belief that nondeterministic outputs can't be validated under traditional GXP standards Because if the same input produces a different output, how do you prove compliance? And here's where the mindset shift helps.

Think about how we validate during a drug trial. Like we don't test every patient individually, but instead we sample a population and prove the result falls within an acceptable band. The same logic applies to AI. The unit you're validating is the model and the outputs are statistical, not binary. The fourth challenge is the disconnect between quality and development, And the fifth being the fact that there is a team layer which compounds all of this.

Developers are moving fast in GitHub and GitLab, while quality is managing validation in PDFs and Excel. Evidence fragments across the disparate tooling, and nobody owns the connection between any of it. This is already hard, and AI is what tips it over. Okay, so let's make the switch from this is hard to what does the regulatory environment actually expect of us. As you can see, there are three frameworks which govern this space.

The first is the FDA guidance on the use of AI to support regulatory decision making for drug and biological products. The second is the FDA's CSA guidance for production and QMS software. And the third is GAP5 layered with ISO thirteen point four five. Most teams treat these as three separate compliance programs, but they aren't. They overlap on ten common expectations.

The frameworks differ in language, not in substance. And here are the ten. I'm not going to walk you through every one of them, but there are three that we have called out, specifically because these are the ones that we see customers struggle with most often. The first is change control and impact. You'll see that as number three.

This is the one that AI completely changes. If a model updates and you can't trace which validation records are affected, you don't have change control. You have a folder of records that may or may not still be accurate. The second is the audit trail and e records, which on here is number four. This is where screenshots start to fall apart, and we'll be coming back to this later.

The third is risk proportionate testing, which is number five. This is where most teams overvalidate the low risk stuff and undervalidate the high risk stuff. And we just talked about that on the cost slide, our other slide. If you build your validation around these ten expectations, you don't need three separate playbooks. You need one well designed system that satisfies the overlap.

For most non product software, that's the more efficient path. For AI enabled non product software, it's really the only path that holds up at scale. It's worth pausing here on this deep dive slide due to the most recent and most directly relevant piece of guidance we have. This is the FDA's February twenty twenty six guidance on CSA for production and quality management system software. The first, there are four main points to this and how you would operate.

The first is to define the intended use because that's what will determine the validation depth. The second is to target AI specific risks like bias, drift and lack of explainability. Third is to trace validation evidence across the model life cycle so that monitoring and accountability stay connected to risk. And fourth, which is the foundation of what we're talking about today, is to take a risk based approach across the full model life cycle where validation evidence, monitoring and accountability all scale with risk, not just at release, but across the entire life cycle. Validation depth should match the potential for unintended outcomes, and the FDA itself is saying so.

Risk proportional validation isn't a Ketryx idea, it's where the agency is steering the industry. The era of validating every internal tool to the same standard is closing because of risk proportional validation. So how do you actually take a risk based approach? The bullets on the left lay out the principle. Validation activities scale with actual risk to product quality, patient safety, and data integrity.

And rather than validating every system component to the same standard, you focus your resources where the risk is the greatest. The goal is to improve efficiency and validation while making sure you stay compliant. It starts with a system assessment. For every tool, you identify what it actually impacts, Ask yourself what decisions the tool informs or makes. If the answer is nothing meaningful, you're probably looking at a low risk utility.

Then ask whether the output is used in quality critical processes like risk assessment, CAPA, or verification. If the answer is yes, your validation depth needs to scale up. And finally, whether the tool's output enters the regulated enters regulated records like the DHF, B and B or CAPA, because that's the highest risk version of yes, and it needs to be flagged and gated. From there, you do a risk assessment where you identify the you just defined. That includes things like data quality, bias, drift, and overtrust, which are the AI specific ones we'll come back to.

Then you categorize and the output is a risk score that tells you where this tool sits in your portfolio. And finally, you assign a risk level. Low risk includes things like meeting schedulers, training records, and view only SOP repositories, and you don't need a two hundred page validation package for any of those. Medium risk covers nonconformance routing, CAPA workflow automation, and complaint logging, and those you want to treat seriously and document well. High risk covers things like automated quality defect detection, a BI system feeding quality disposition, or an environmental monitoring system.

And this is the category where your validation effort actually lives. By scaling your validation activities based on these potential risks, you don't slow down innovation for low risk utilities and you keep your high risk systems fully compliant. Before we get into the mechanics, it's important to acknowledge that for large organizations, adopting a risk based validation approach isn't just a technical change. It usually requires changes to process, changes to mindset, and often better tooling. On process, your validation SOPs were written with when CSV was the only option, so they weren't designed for a CSA aligned risk tiered approach.

Changing them means changing approved procedures, and that's a project in itself. On people, a lot of QA teams grew up validating everything. So the mindset shift to we don't validate this the same way we validate that is often harder than the technical shift. On tools, legacy ALM and QMS systems literally don't support risk tiered validation as a workflow. If your tool can't model differential rigor, your team can't apply it.

None of that means this is out of reach. It just means you need the right approach to help you work through it. With that context, let's look at how to actually apply a risk based approach to validating AI enabled non product software in practice. That we've walked through the risk based approach and named the organizational barriers, there are the three operational challenges that show up at the work level. The first is scoping.

And the question we hear is, are we evaluating the right elements? Most teams either over scope or under scope because intended use isn't defined. What works instead is to treat scoping as a one time decision per tool grounded in intended use with a documented rationale you can defend. The second is evidence. And the question is, where are screenshots still showing up?

Manual testing produces images of a passing test, and that's not an audit record. The fix is to use structured, queryable, audit ready records stored in your QMS rather than a shared drive or someone's inbox. The third is change management. And the question is, are model updates triggering revalidation cycles that you're not tracking? Most teams find out about a model update in production.

What teams should do is tie revalidation triggers to version releases instead of calendar dates. So you revalidate when the thing actually changes and not when the quarter ends. All three of these trace back to the same root problem, which is processes built for deterministic software are being applied to non deterministic systems. Let's start with scoping. Let's make this one concrete by tracing one specific risk end to end.

Our use case will be a complaint triage agent and the risk is missing a real reportable event. First, we'll define the intended use. The intended use is to classify incoming complaints by severity and flag potential reportable events, while preserving traceability back to the source intake record that tells you the decision being made, who reviews it, and what record it feeds. The risk that matters most, as we identify the risks, is a missed reportable event. The model might label a real medical device reporting as non reportable, downgrade severity, or miss a criterion buried in the complaint text.

Next, we select metrics. The metric is recall on confirmed MDRs, with false negative rates as the primary KPI. From there, we'll choose the monitoring strategies. The monitoring strategy is a sample review with recall gating. Every flagged event gets a human review.

And a recall floor has to be met on every release. If the floor is breached, the release auto blocks and drift is checked on every model version. The mindset shift is that AI operates within a band of accuracy much like humans do. The goal is to define that band and catch any drift outside of it. And finally, documenting the rationale.

You document the rationale in one place, covering risk, the metrics and monitoring. That's what you would bring to a notified body, and the same logic applies to every other risk on this tool. The core shift is automation. We're talking about automatically generating evidence of testing straight from source tools like Git instead of having engineers take manual screenshots of automated tests just to prove they ran. Looking at the V model, every single artifact required for a sixty two thousand three hundred and four style validation package already exists in your day to day development tools.

Your use cases and system requirements live in issues and tickets. Your commit messages are tagged to requirements. Your design rationale lives in pull requests, and your architecture docs are sitting right in the repo. On the other side of the V, your unit tests, integration tests, test reports, and version change logs are already running. The reason teams still resort to manual screenshots is that no single platform spans the full V model.

Git holds part of it, Jira holds another piece, and your test platform holds a slice of it. Without a unified system, traceability has to be managed manually, and the final evidence ends up as a folder of static images. The problem is that tools don't interoperate to prove compliance automatically, and here's how we change that. We said earlier that change management needs three things, which are tracked versions, monitored drift, and requalification cadence. This is the lifecycle that delivers all three.

There are two layers here with a model first and then the data flow that feeds it. So what does AI change management look like when a model needs to be updated? You start with a validated model. When an update is needed, you retrain, which generates a new model. Before that new model can be released, it goes through an acceptance criteria impact assessment, and that's where you catch drift and degradation.

If it passes, it moves to release for system revalidation. This is the same statistical logic from the drug trial comparison earlier. Running this impact assessment confirms that the new model still performs inside the acceptable band you defined. Now to the data side. New data comes in and goes through data analysis, where you ensure data quality and data segregation by separating training from test data.

From there, the data splits. Your old plus new training data feeds the retrain step. Your old plus new test data feeds the impact assessment. Throughout the entire loop, documentation is automatically generated. Every time a model is retrained and deployed, you have a traceable audit ready record for regulators.

Here's a quick example case study. And remember the complaint triage agent from our scoping example earlier? This is the real version. This customer is a global leader in clinical research digital platforms, which is the infrastructure that pharma and medtech manufacturers use to run complaint trials. Their complaint Trios agent is AI enabled non product software.

It influences GxP decisions, it sits in their regulated workflows, and it requires validation. QA and RA were spending weeks per release manually compiling that evidence. So the bottleneck wasn't development, it was compliance. They applied the risk based approach we just walked through, covering scope, evidence, and change management. And they used Ketryx to automate the documentation.

A multi week manual effort became a background process. The results, documentation time went down by eighty percent, change management became connected across systems, and redundant requirements were resolved automatically. Automation strengthens compliance when it's applied with the right structure. And James is about to show you what that looks like inside the platform. I'm going to hand it to James now for a live walkthrough of that complaint triage agent we talked about earlier.

Something to watch for as he goes through it. This is Ketryx validating Ketryx. We validate our own AI agents and our own integrations against sixty two thousand three hundred and four, thirteen forty five, twenty seven thousand and one and fourteen thousand nine hundred seventy one. So what you're about to see is the same playbook we'd hand to you on day one. And with that, James, I'll let you take it from here.

Awesome. Thanks, Rich. And just a quick frame as I share my screen and open the platform because most of you will not have seen Ketryx yet. The way that we work is item based. So an item is going to be the unit of work that Ketryx directly applies your quality rules to.

Items come directly from your source systems. That's gonna be things like Jira for requirements management and Git for tests and automated testing. So instead of copying and pasting a requirement from Jira into your THF or DDF or taking a screenshot of an automated test result from GitHub, those will actually show up in Ketryx as items. And one more disclosure as I demo, this is a demo environment. The data is gonna suspiciously look like real customer data because our assistant is good enough.

So it feels that way, but it's not. So if anything looks close to home, that is by design, and that means that it's working. And so with that, let me show you a complaint triage agent from end to end. I'm going to start on the Ketryx platform on the all project screen. I wanna highlight three projects here.

That's going to be two applications, one AI health application, our complaint processing application, and then the third one is going to be the actual NPS Gen AI model. What I wanna call out here is that for each of these three platforms or projects, they're each on a different version. So two dot one for the AI health application, two dot o for complaint processing, and one dot two for NPS. The AI platform lives in its own version versioned independently. The incremental use cases like the AI health application and complaint processing are actually built on top of that platform.

The key value here that we really see Teams unlocking is because it allows you to validate the underlying model once and then actually reuse that validation evidence in any project built on top of that. You don't have to redocument how you tested and trained your model. You just have to reuse it in your process. You can also see that each of these projects are connected to those source systems like I mentioned earlier. We are connected to Jira for requirements management and then Git for testing.

I'm now going to take us into the complaints processing application. Here, we are going to be seeing the all items screen in Ketryx. Again, items are things like requirements coming from requirements management systems such as Jira and then tests coming from systems like GitHub. You can see that we are mid flight through our release of this complaint processing agent, our different item types of requirements and tests, and we're actually gonna be using AI to accelerate our validation process. The way the first way we're gonna be doing that is gonna be using our Ketryx assistant.

So I'm going to open up our assistant right here, and I'm just going to ask the assistant to tell me what's changed and been removed in this version. And so what the agent is doing right now, it's actually going to be reading this assistant is going to be reading all of those items from the source systems. It's going to be understanding the context of your project, your QMS documents, and all of your items. So now when you're talking to it, chatting with it, it has that full context, and it's able to take your semantic questions and find those answers. It's going to identify what's changing and what's been removed.

And here we go. It's looked at what's been changed and removed in this version two compared to the previous version. And we can see that there have actually been fifteen items changed, so six requirements, one software item specification, a risk, and then seven test cases. Immediately using the Ketryx assistant, you're able to get up to speed on where you are in the release and the project. Another question I might wanna ask it is, you know, tell me about the anomaly being resolved in this version.

Now the assistant is gonna go one step further. It's going to take my question, which is kind of a semantic English question, and it's going to actually find the anomaly that's being resolved, and it's gonna go beyond kind of title search match. It's going to go in and fully understand the context of that anomaly, the details, the metadata, and even the linked items. So here we can see that the anomaly is going to be highlighting a severe adverse event recall. Can see what happened, the impact on the system, root cause.

And then down here, we can see the traceability. So the assistant can actually identify what items this anomaly affects and impacts. So it's allowing me to understand directly what's impacted by this. And finally, I'm gonna ask it to let's go show me the tests testing requirements affected by this anomaly. This is gonna be become a multi hop query for the assistant.

And I just wanna wanna highlight here that, again, this is going beyond, like, a Jira query language or an Azure query language. This is truly it's taking my semantic question and understanding what that means in terms of the linkages and the items affected. I talk to teams like this every day, every week, and when they see the assistant and the power of the assistant, they're truly mind blown because queries just like this one take them hours, days, or maybe they're not even possible with the limitations on those other query languages. And that's why the power of the Cataracts Assistant is so, so, so powerful. Here we can see that the anomaly affects one requirement, which is verified by a single test case.

So let let me understand which tests are covering the changed requirements so I can verify that our anomaly in this version is actually being resolved. Now we've looked at the assistant for kind of a, like, a reactive way to understand what's changing our project to make sure that we're up to speed. Another way that we can do that is what we like to call proactive compliance with our AI agents. In our in complex AI systems with tons and tons of requirements and specs changing perversion, tracking things like conflicts or redundancies between requirements is nearly impossible. Our Ketryx agents, they run proactively in the background, so it keeps your project compliant in the background.

You don't even have to think about it. I'll navigate to the Ketryx agents tab right here. You can see we have a requirement conflict detection a agent. What this does, it's able to run manually with this button here. It's able to run at a specific, specific time, or it's able to run when a new requirement is being created or updated in that source system.

What this means for your teams is that every time an engineer or product manager changes a requirement in Jira, updates a test, etcetera, this agent will run, check the whole project, and identify any conflicts or redundancies before you're even able to look. So as soon as you log on in the morning, as soon as the requirement has been updated, you're getting that compliance and you're getting that update. The agent will run, and it will actually give you suggestions on what to do. So here we can see one of the suggestions being there are duplicates or overlapping severe adverse metric requirements. We can see the agent found that this requirement duplicates the already related one, and it's suggesting that we edit the requirement wording.

I'll go ahead and I will review that. Something to highlight here is that with our AI tools, so both our AI assistant and also with our AI agents, is that we always like to keep a human in the loop. So we like to keep a human subject matter expert always be the one to provide that final review. Our AI assistant and our agents will never actually create or edit or change items without that human subject matter expert working. So really the goal is to help those subject matter experts reach their decisions as fast as possible.

So you can see here that the agent went through, and it is gonna be suggesting this change. We can look at the change, scroll down, and see here the description of the requirement, original value, new value, and then the actual red line changes. And so if we were to see if these changes or accept these changes, those changes would actually be pushed out to the source systems of Jira in this case. I'm now gonna pop over to Jira and talk about how we help developers remain in their tools in Jira and help Jira become compliant. Something that we like to talk about in Ketryx is that we'd like to keep developers in their preferred tools, such as Jira and Git, and continue continuing their same workflows.

We don't wanna disrupt their everyday workflows. We just want to make compliance something that happens in the background and something that is a byproduct of the work the teams are doing in their systems. What that means with Jira is that we've added a few things to make Jira a little bit more compliant and work with Ketryx. If we scroll down on this anomaly right here, we'll see a few things. The first one right here is going to be the Ketryx traceability widget.

What this does is it allows you to see one level up and one level down of traceability and linkages right here in Jira. This allows developers, product managers, folks working in Jira to fully understand and have that visibility into what their items impact or affect as they are working on it. They don't have to look across separate tools or look in a spreadsheet and do some matching just to figure out what their items impact. They see it here as they're working. The second thing I wanna impact or highlight right here is going to be the Ketryx approvals widget.

We put a twenty one CFR part eleven esignature widget directly in Jira. This means that your quality team is not gonna have to chase down engineers for sign offs because they're gonna be able to approve these items at an item level right where they are already working. Again, just to highlight, these are fully customizable to your workflows work workflows and your use cases. We wanna make these work for you, and we'd like to make sure that no one is changing their workflows. I am going to go through and approve this item.

And with that, we can see that the anomaly has been closed and is in a controlled state. And something to highlight here is that it actually creates a controlled record in Ketryx. So if I open this up in Ketryx, we're gonna be opening that same anomaly in Ketryx. Because we're gathering those approvals and we're and we're collecting those clicks and that evidence, we are essentially creating a living audit history trail. So every edit, every approval, every rejection is all gathered here in Ketryx right here, and we can see every single piece of evidence.

You can see the changes between items. You can see the difference. And, it creates that living piece of audit history. Now that we've looked at local traceability and item history, I wanna show you the full traceability matrix. We like to kind of reframe what traceability is with Ketryx.

I think for most teams that I talk to, traceability is a few things. It's either a stale artifact that's created at the end of the development process that's an Excel sheet because the FDA FDA requires it, or maybe it's a Power BI dashboard that was created and it takes hours and hours a week to maintain. And as soon as a new version is published, it's actually outdated and obsolete. We take that, and we turn that on on its head at Ketryx. What you're looking at right here is a live knowledge graph across your entire systems.

So as your team does work in their tools in Jira and in Git, this traceability matrix is updated in real time. Something to call out here is going to be these middle columns of this traceability matrix. These are actually going to be verification tests from the separate NPS AI model that we talked about in the beginning. These tests and these evidence is going to be reused in this traceability matrix. So we're not just validating these columns are not just validating this specific use case.

We're actually showing the underlying documentation for the platform and for the model. What this really unlocks is that teams don't need to redocument that model. They can reuse this evident evidence. They can reuse this verification testing, and they can reuse this validation. I also wanna highlight automated testing.

I believe the stat that Rich mentioned was at the use case level, something like eighty percent of validation testing can be automated. So actually integrating and interoperating into where that automated testing is being done and automatically pulling that into documentation is so critical, and it can really speed up and accelerate Teams' development. And we've seen that with tons of customers. Something else that this traceability screen can really enable Teams to do is live change impact analysis. In the past, we hear stories from teams that when a change is made, it takes weeks, maybe even a month to understand that full impact.

Multiple teams have to meet. Subject matter experts have to get in the room, go through every single item, and understand that impact. But because we have this live knowledge graph, it's never been easier to do that change impact analysis. We understand how every single piece of the puzzle fits together. So if anything changes, we can do that impact analysis instantly, and it is a huge unlock and accelerator for Teams.

Finally, I'm going to navigate to the release process. So once we're ready to release our version of this software, we'll go to the releases screen over in Ketryx. We're gonna go to this version that we've been working on. And here, we can see a high level dashboard of where we are in our version in our project. We can see the number of controlled items, uncontrolled items, new items, change items, as well as a release checklist.

We can see here that we actually have three uncontrolled items. As we click on this, it'll actually take us to those items themselves. We can see here that it's a few requirements. I can go through here and actually approve those right now. Now that I have gone through and approved those items, if we go back to our release checklist, we can see that the checklist has been released.

And what we can do now is regenerate our release documents. Where this has all been building is that as your teams are working in Jira and in Git, Ketryx is pulling in that evidence and creating those that release documentation. Because we're pulling in every stroke, every edit, every approval, it comes time to generate those documents, all it takes is one click, and then all of your release documents can be generated. We can see here that the system requirements specification was just generated. I will open it up, and it's gonna be a Word document.

And you can see that it will contain all of the requirements from your source system, Jira in this case. Something here to highlight is that these are fully customizable to the way you work, both in content and in format. So we are able to and Ketryx is able to process templates of any format that you want depending on how you submit your documents and is also fully able to customize the information that comes into your documents. Here we can see actual system requirements are being pulled in. This is an item that comes directly from Jira and the source system and was pulled directly into Ketryx.

Now that we've generated our documents, we're able to go and actually approve this release. We're We're gonna click here, and we're going to approve this release with my esignature. And we'll see that the release has been approved. I was just walk just walk you through an end to end complaint management agent from the verification design to verification to traceability to release. Thank you all for watching, and I am more than happy to take any questions.

Great. Thanks, James. There are a few questions that had come through. Maybe the first one or the one that I see from Diana was around EU MDR. So, we have the guidance from FDA from earlier this year on CSA, but are you familiar with the EU MDR?

If not, we can also take that as an action. Yeah. It's a good it's a good question. And we are we are familiar with EU MDR. We work with EU MDR.

And what's nice about Ketryx, and I think especially our documentation and our templates, is that we can have specific documentation, specific templates for US guidelines, for UMDR guidelines. So whatever whatever needs to be submitted, wherever needs to be submitted, we can help with that and answer that. Great. And then there was another one. Maybe I'll just, I'll just read it off.

I did respond in the the Q and A section, which for folks, if you're not finding the Q and A section, there's the more of the three dots on the bottom. Was down under there and I had to add it to my view. But in the example, James, as you were going through and looking at the agent, question was in that example, was an agent editing a design requirement? How do the agents comply with regulatory requirements for design reviews? I provided a response, but if you wanted to also provide a response on that one, I mentioned the human in the loop nature and still, you know, requiring your to follow your SOPs or your QMS.

Yeah. I think that yeah. You you nailed it, but then there's a little bit more color is that out of the box, we provide some validated agents. And so this requirement conflict and redundancy detection agent has been validated. It's been tested.

We've gone through that. And then again, whatever the agent findings are, there's going to be that human in the loop. So we're not going to be replacing anyone's subject matter expert guidance. We're simply gonna be getting, you know, to eighty percent of the way there and having them review and approve that versus starting from a blank state. Perfect.

And there was a follow on from Elizabeth on that with the, in the context of human in the loop. Can you include multiple users in the loop defined by job role or group? It's a good question. I I believe currently the for for agents for assistant findings, it's gonna be one human, one approver. However, what we can do with human in the loop and for approvals is that you can define custom approval groups for specific items and for specific source systems.

So if you want, for example, say, your requirements that are coming from your requirement management system for Jira need to be approved by multiple groups such as project managers, product managers, quality managers, regulatory team. You can define that. Perfect. The next question was around from an anonymous attendee. Can this platform also be used as an EQMS for SOP updates and record generation like CAPAs, management reviews, internal audits, etcetera?

Yeah. It's a good it's a great question. And, yes, it absolutely can be used for an EQMS. In fact, we see a lot of customers using it for both an ALM and an EQMS and seeing the benefit of that because their quality events, their QMS documents are actually going to be in the same system as their design controls. And that way you can truly get continuous compliance with your QMS documents and your design controls.

And then also you can manage all your quality events in one place. And we see customers getting a ton of value out of that because I think with QMS, there's a ton of systems. There's a system for Kappas, for complaints, for for NCs, NCPs, scars. We bring them all into the same system, and it lets our AI, again, have that full context right across them and lets you speed up your quality management activities. Great.

I see one more from an anonymous attendee. When you talk about statistically validating these systems, how do you define your passing criteria and sample sizes, as well as how do you create and justify appropriate validation data sets? Yeah, that's a good question. And Rich, feel free to add any more color. But I think that going off the CSA guidance is to take that risk based approach.

So think that you've again, you need define that intended use. You understand which risk category you're in, low, medium, high. And then from there, that will, I think, allow you to define the data set, acceptance criteria, and then you go from there. Yeah, that's perfect. I think the clinical trial analogy is a really strong one, and we don't do full population test results.

We do gather all that data, you are aware, but it is based on the sort of what's acceptable. I think that acceptable is part of what we're going through with identifying that risk and having it be risk based. Hopefully that helps. A good question. The next one from an anonymous attendee: What is the best revalidation cadence?

Yeah, this is this is a good question, and I think that this is really kind of where we we can talk through Ketryx and how that helps you with that. I think that what we've seen for revalidation Ketryx for AI or cadence, sorry, for AI models is when there's a new update is to do that revalidation. And that's why automated testing is so, so important because you set that acceptance criteria, you run those automated tests. And if everything passes, you can just reuse that validation that validation effort and those validation documents right here in. So instead of having to maybe revalidate your model every time it changes for every single application, you just revalidate it one time and you can reuse that evidence.

Very good. The next question I see is how do we handle reapproval processes? How do we handle reapproval process in case of incorrect earlier approvals on Ketryx? Yeah. When if there are incorrect earlier approval approval processes or incorrect earlier approvals, you're able to kind of reopen or reopen that that item or reopen whatever you're looking at.

And then you can go through the correct approval processes. But I would also like to say that the approvals are going to be deterministic, so we have to define those approval groups. So, actually, Ketryx will not let you approve an item until it's gone through each of those deterministic checks. Great. Next question.

This is great, guys. Can you include multiple users in the loop? Oh, I think we asked this one already. And the response was there's there's a place for agents right now, I think, is one to one, but there's a place where you can add in multiple approvers when you get to the actual item that's being changed. So it's a little different way of thinking of it.

And then a question here from Eric. How is it satisfying part eleven? I didn't see the requirement for providing at least one more token password for approval after a continuous login. Maybe it didn't pop up on on the screen share another screen share, but, typically, when you do your your part eleven approvals, it'll it'll it pops up. So it popped up for me, and it's either a touch ID or it's a password, but it must not have popped up on the screen share.

But that's how we do part eleven approvals. Great. Thank you. And then another one from an anonymous attendee. You said the Ketryx could be used in EQMS.

I'm wondering if there will be support to set that up. Oh, is this from you, Nima? Your name was in parentheses, so I just read it off. But is there support to set that up, set up the EQMS piece, maybe is the question. Yeah.

And, yes, abs absolutely there is. And we have a dedicated team. This is gonna be Richard's team, the delivered success team that's dedicated to helping customers get online and get value and get support. And they are they do do this every day, best in class, best in the world, and can absolutely help you set this up with no friction. Yep.

Very good. Okay. I don't see any other questions. I will talk slowly until any other questions come through. Otherwise, thank you all so much.

Just doing one last look here. Yeah, thank you all so much for your time today, and thank you RAPs for partnering with us on this and for your participation, and hope you all have a great rest of your day.