---
title: "Breaking the Change Impact Assessment Bottleneck with AI"
type: webinar-transcript
publisher: Ketryx
source: "https://fast.wistia.net/embed/iframe/9t8gngsfuf"
content: auto-caption transcript, proper-noun corrected
---

# Breaking the Change Impact Assessment Bottleneck with AI

*Ketryx webinar — transcript of the recorded session.*

[▶ Watch the recording](https://fast.wistia.net/embed/iframe/9t8gngsfuf)

---

joining today's session on breaking change impact assessment bottlenecks with AI. We'll get started in just a minute when more folks join online. I'm personally really excited about this topic because change impact assessment sits at the center of pretty much every single regulated product development project. While we're waiting for folks to join, Bailey, would you mind sharing what you see that you work with, struggle with with change impact, and where this impacts their workflows today? Yeah.

Absolutely, Megan. I'm also really excited to talk about this. Both of us have some experience in systems engineering, which, of course, we'll talk a little bit more about once people join. But, really, when I saw it being done then, I did that for about three years, and it was all manual. And so I'm really excited with the introduction of AI to help automate these processes, bring information to teams a lot faster, and not have to go to from SME to SME to get the right information.

So I think it has, it's a very powerful tool to transform the way that teams are doing change impact analysis. Totally. I completely agree. In my prior experience, and I see this with a lot of teams too, change impact assessment is manual. It's fragmented, and it's difficult to keep up with, especially as systems become more complex.

So let's dive in and unpack what there is to share with some solutions that Ketryx has to offer. So before we get started, a few housekeeping items. First, we will send out the recording and the slides after the webinar. So feel free to sit, soak in the content. No need to take screenshots or worry about missing anything.

It'll be there for you. And throughout the webinar, if you have any questions, feel free to drop it into the q and a at any point in time. We'll keep an eye on it throughout the session and save time at the end to address the questions that have the most common prevalence. And finally, we'd appreciate your feedback. At the end of this webinar, we'll be sharing a short survey.

It helps us shape future sessions and make sure it serves you and your teams. So let's talk a little bit about Ketryx is an AI native compliance platform purpose built for regulated teams to deliver safer products. It acts as a layer unifying the data across your development tools, your Jira, your JAMA, Google Drive, to create a single source of information across your team's organization and then ply your apply your quality management procedures in those tools directly. With that data, Ketryx also automatically generates documentation as evidence of compliance and anything that you might need to meet regulatory needs. And with AI, we can automatically trace requirements and assess change impacts, validate test coverage, imply compliance risks in real in real time.

So, like, what does this mean for you and your teams? For quality, this means less manual chasing of evidence and more confidence in audit and submission readiness. For r and d teams, it allows them to spend more time building and less time documenting. And for systems engineers, instead of developing evidence after the fact and chasing teams, it embeds the compliance directly in the tools and allows you to focus on critical work as opposed to manual and automated tasks. So we'll quickly introduce ourselves.

My name is Megan Menino. I am a technical product marketing manager here at Ketryx. Prior to my role, I spent over six years in med tech development primarily as a systems engineer for cardiac devices, most recently at Abbott supporting class three implantable medical device development. I love to take those insights in my current role to help convey the value of Ketryx and influence our product road map to make sure that our solutions meet our customer needs. Bailey, would you mind sharing a little bit about yourself as well?

Yeah. Absolutely. Hey, everyone. I am Bailey Cantor. My background was originally in physics, and then I was a systems engineer in the defense space.

So that's really where I started to see the challenges and the intricacies behind all of the different, teams that systems engineers work across from requirements to testing to risk management. And so I saw that intimately at Raytheon. And then I was a back end software developer and scrum master at Amwell. And now I lead our solutions team here at Ketryx where I'm very passionate about how teams can use AI to make the change impact assessment more efficient and really to automate all of the processes that go into product life cycle. Thanks, Bailey.

So why are we here today? The volume of change in regulated development is exploding. There's a statistic from GitHub that ninety two percent of developers are using AI assisted tool. So what does this mean? This means there's more frequent changes across systems that are becoming more and more complex and are interconnected with each other.

When a change is introduced, it doesn't stay isolated. It impacts requirements, design, test, and risk controls across an entire system, quickly cascading. Small line of code can become a big exercise very quickly. And for many teams working with us, it's our understanding that a change impact assessment can take six or more weeks just for a single simple change. And this gap has real consequences.

When we see software related recalls, they're happening every single month, and they're often tied back to software missing impacts or verification. The challenge is both the growing complexity of systems and the fact that change impact assessment is still largely manual and fragmented across tools. As development accelerates, that approach doesn't really scale. Manually going through tools and figuring out where a change lies doesn't scale when your changes amplify. And that's really where the bottleneck we're here to discuss is lying.

So for today's agenda, we will cover common change scenarios and how the change impact propagates across the system. Then we'll explain why manual approaches don't scale in complex environments. We'll impact on how time is lost and risk shows up in a typical change impact assessment. And finally, how to scope changes in minutes instead of weeks using Ketryx. So let's take a minute to baseline.

We'd love to go deeper and get a quick sense of where everyone's coming from. One of the biggest variables we see across teams is how they perform change impact assessment today. Some teams have a formal, well documented process, while others rely more on tribal knowledge and ad hoc reviews. And most fall somewhere in between. So let's take a moment and answer the poll.

How does your team currently handle change impact? You can open up the polls tab and select the option that fits you best. As folks are filling this out, Bailey, from your perspective, where do you see teams struggling most with change impact assessment for today? Yeah. I think it's a very difficult process just in its origins.

You have to really think about all the different components in your system that you're affecting. And, previously, this was done manually. For new engineers who are often tasked with doing change impact analysis, this can be very difficult. Like I mentioned before, having to go to a SME and get that knowledge of why did we make this change? Why did we introduce this requirement and not consider, you know, this impacted risk?

Those types of things are are typically, you know, in the the SME's head and not on paper or documented. With the introduction of AI, however, this becomes a lot easier to get that knowledge into the hands of users quicker. And, of course, this issue compounds when you're working across many different components of your systems and when you have teams working across different versions. And so trying to understand if this applies to a specific version of a requirement can be very difficult alone just trying to manage, all the different versions that teams are working on. So it's just an added complexity.

And so for teams that are doing change in analysis for any device, really, or any software, it's difficult. And then you add on as well the regulatory standards for teams that are working in the medical industry where you're not just doing a change effect assessment, you're also doing a risk assessment. So there are all these compound compounding effects that really make make a change impact really difficult for teens. Totally. And I would agree.

And even when I've done change impact assessments with current AI tools like Copilot, I find that it lacks the context to really provide actionable insights. manual reviews can be time consuming and laborious. But, also, when you put on AI tools that lack the context, it doesn't really provide concrete recommendations that teams can take action on. And it looks like the majority of folks are use sorry. I gotta extrapolate my screen. There we go.

It looks like most folks are doing manual reviews like you had mentioned, Bailey. Some people are using partially automated processes, but a lot of manual work. Yeah. And I think that makes sense too. Right?

I think systems engineers, we love our Excel spreadsheets. We love having a logical order of things. And so, you know, when you're using all the tools that teams are working in, like software development tools, it can be very hard to jump across all these different tools to gather that information. And so you often reside to sticking to Excel to track and pull that information from those fragmented systems. And so I think that's very common.

I'm curious to know how teams are, you know, maybe even using AI or trying to introduce AI into some of those manual workflows. Totally. Okay. Thanks, folks, for answering the poll. Let's go over some common scenarios where change impact occurs.

We'll start with the rogue change. An engineer updates something, whether it be a requirement or code or a risk, without any formal review. And then downstream, it starts to propagate. You learn about it at a late stage, and it should have triggered or changed impact, but it's catching up with you at the end. And nobody really realizes it until a drift occurs or a test happens, and you learn something that is surprising.

Then there's the mid development pivot. You learn something from a test or maybe a market insight. A competitor comes on the market, and you gotta change what you're working on. Development's already in motion. Requirements, design, implementation, and tests are already connected, but you need to reevaluate everything and figure out what changes need to be made and what that means in terms of your project, product, and team.

And then lastly, there is the post market fix. So something happens in the field. You get a field issue, customer feedback, or CAPA that requires a change to a released product. Now the stakes are higher. In addition to making changes and cascading across, like, different tools, there's also the regulatory scrutiny that comes with making a change with a product that's in the market.

And it needs to be clear what was impacted and why the system is still safe, especially if it's a safety related issue that's prompting it. And then when we look at AI, AI assisted coding is becoming more and more prevalent. As we talked about before, about ninety two percent of developers are using AI to assist with their software development, which means there's faster iteration cycles and there are continuous updates, which are increasing the volume and velocity of change. And with manual tools, it's really difficult to manage all that. So there's more changes.

It's moving faster and across more complex system while impact assessments are really manual. So what does this actually mean? When as we talked about before, there are software recalls that are occurring on a monthly basis, and the majority of them are related to software impacts that are missed or verification test gaps. Eighty two percent of software recalls trace back to design issues, And each change impact takes about six weeks according to most of our clients. Now this can cascade over time to be very expensive.

A major quality event can cost hundreds of thousand hundreds of thousands or millions of dollars. This isn't a one off failure. Small gaps count compound. And when impact assessment is manual, those gaps are easy to miss until they show up in audits, delays, or recalls. So the challenge is how to assess impact continuously and not retrospectively. quick thing I just wanted to add to that.

You know, I intimately saw that as my time working in defense where I had to we were actually using DOORS for our requirements management tool, and we had a change come in to requirement. And I really had no clue where to even start. At the time, we had no really great search capabilities to go through hundreds of thousands of requirements. And so I really was going to each person to figure out what did what did this requirement mean, often sometimes finding a requirement that wasn't written in a way that was testable. And then that would delay my my investigation even further.

And so I can intimately remember, you know, spending probably two months just tracking down one single change across different tools, working with my scrum teams and on the software teams to try and figure out what the changes to the software needed to be made to implement this change. And so when we think about speed of making these changes to the software, that really wasn't happening in the defense industry. And I bet many teams are are struggling. I do work with teams that are struggling to match that same speed of changes with how often you need to update the code and how often you want to find a fix or fix a bug. And so that really does delay the amount of time that you can get that fix out to your to your teams that are using it, whether that's a medical device and you're trying to provide life saving care.

A lot of that time is just spent doing these manual tasks, like finding what it impacts. And maybe it's even just a few days are spent fixing it in the code. And so I really wanna shift the way we do that, where we spend the same amount of time, if not less, fixing the code as we do to find it. So just wanted to highlight that. Totally.

I appreciate you jumping in and mentioning that. In my prior role, I used to talk to a lot of clinicians, and one of the hardest objections I had was my cell phone software updates, like, on a monthly basis. Why does it take ten years for you to update your software? And, honestly, it's a really hard question to answer. And with tools that help automate these processes, I think it's a solid way to allow us to release sophomore work quickly and get products on the market that are safe, effective, and meet the evolving nature of clinical needs.

So where does the time go? Let's start from the beginning. So many of you might have gone through this process where a change is proposed. And maybe that takes a few hours to figure out what the change is, where it's coming from, and take a decision to act on the change. Then you gotta identify the scope.

So, like, what does this change mean in the context of everything that we're working on? A variety of subject matter experts are deployed to go figure out what does this mean in the context of our code, what does it mean in our requirements or risk, and eventually come back together to discuss what this means. And after getting that initial assessment of what it means, then there's tracing the dependency between things. So if I update my requirements, does that impact my risk controls? Does that risk control require me to repeat a test?

All these things are related together. And due to the manual and fragmented nature of tools that are used during development today, this can be a lengthy process. Here, we have it listed as one to two weeks, but I've been on teams where this alone takes four to six weeks depending on the complexity of the project and the changes that you're looking at. Then there's the assessment of the impact. So as Bailey had mentioned, there's this downstream conversational aspect of understanding the why behind things.

And sometimes that relies on talking to subject matter experts to really dig in and understand, like, why did we decide to do this? Where was this coming from? Especially when there's not rationale or in the event that subject matter expert isn't there anymore. Maybe it's additional research and repeating the work that was initially done. Then there's documenting and acting on the change.

So taking the effort to move on the changes that have been assessed, been traced, and acting on them. And then the documentation workflow, which three to five days can be fast in my experience depending on the team that you're working with. But this final stage of, like, approving and closing everything ties everything together. And as you can see, there's a highly manual nature, especially with tracing dependencies, assessing impacts, and making the changes due to the disparate nature of the tools that folks work with, resulting in six or more weeks of time just assessing a change. So let's talk about how AI assisted change impact works.

So first, we have a change just like today that comes up from anywhere that we talked about. It could be a road change. It could be a scope change. It could be post market. Then an agent is deployed with the context of your design history file, the project workspaces, everything that's connected from GitHub all the way to your requirements.

And it goes through and identifies the potential impacts, Reviewing traceability. So it looks at requirements and helps identify, okay. Based on this change, do I need to make a modification, or is there an impact to an existing requirement? It provides context of specification impacts. So does this impact a hardware specification or software specification based on the proposed change?

And then based on all of this cumulatively, how does this impact our test cases both for specifications and for requirements? And comprehensively, the agent provides an output to the human in the loop, the expert, for subject matter expert review, always keeping the human in the review with the opportunity to take or leave any of the suggestions and iterate back and forth with the AI. So the AI agent allows you to more quickly focus or you to more quickly understand what the context of a change is and have the opportunity to focus on critical thinking as opposed to manual tracing to get an idea of what's going on. Yeah, Megan. And one thing about this workflow, which I love how it just, you know, lineates the workflow that teams are working on and what is typically a more roundabout manual process.

But one thing here this doesn't show is risk as well. And we can add and have agents also add risks, and do that risk assessment, especially for the teams that are trying to adhere to standards like one four nine seven one. So that's just another added step here that the agent could do. Totally. So let's see an example of what this looks like in practice.

We recently worked with an AI driven diagnostics company to improve their change impact assessment process to be seventy percent faster. At the beginning, when we first started working with them, change impact assessments were laborious, taking up to fifteen days. Their subject matter experts needed to manually review their tools, including Jira and Polarion, to understand how things related, and it resulted in their team moving very slowly on changes, sometimes with critical bug fixes that impacted the field. We implemented an InjectiC AI solution and end to end traceability to understand the context of things. So this AI agent had access to over two thousand requirements and within a few minutes, identified ninety percent of impacted items, including requirements, specifications, and test cases, allowing their team to iterate with the agent to evaluate change impacts.

So what did this mean for them? It accelerated their change management. So they were able to go through their change impact assessment seventy percent faster from a weeks long process to a few days. And these assessments were high accuracy and also provided insights that they didn't even think of before. It about it increased their SME availability as well to work on other high critical tasks associated with actually implementing the changes rather than just trying to get the context of what the change means across tools.

So let's take a look and see what this is in action. Bailey, I'm gonna kick it over to you. Wonderful. Thank you very much, Megan. I'm going to share my screen here in just a second.

I think if you stop sharing, Megan, I can share. There we go. Perfect. I always run into that. Alright.

I will share my screen now, and you just let me know when you can see it. Yep. Awesome. See. Wonderful.

Well, for those of you who are seeing the Ketryx platform for the first time, welcome. This is where we'll be for the next, you know, half hour or so with some time for questions at the end to explore how teams that we're working with are are actually using AI to automate the workflows today and how they use it to do a change impact assessment. Feel free to post your questions in the chat as well, and my team will let us know, and we'll try and answer them live as we go. Throughout this demo, I'll also explain how that case study that Meg just mentioned was actually implemented, and that will be the foundation for our workflow today that we'll walk through. Alright.

So let's jump in. What we're viewing here is our project dashboard in Ketryx where we're managing many of the different products that our team is working on. For the bulk of, what we'll explore today, we're gonna be in this insulin delivery monolithic system, which contains all of our design controls, and we can create our entire DHF just from this project. We can also see the tools that these teams are working in for that creation of the DHF, and that's in Jira and Git. So this team is actually doing requirements management in Jira for their work, but we can connect to many different tools that teams are working in, whether that's JAMA, Polarion, preferred systems, or requirements management tools and test management tools as well.

We're very much agnostic to the tool that teams prefer to work in. We really sit across those tools and bring together that information so you can view it and then run AI across it. What we're doing there is gathering all of that information because having the context is very important for AI, as Megan mentioned earlier. Before we jump into this insulin delivery system, I just wanna call out the use of the systems and systems architecture here, which we won't be diving into deeply. But what we're seeing is this top level irregular rhythm notification system that's referencing a few subsystems or components.

In this case, it's referencing it looks like these two subsystems that you can manage independently in their own Ketryx project. You can have them being released on different versions so that your slowest moving system, maybe that's even your hardware system that doesn't change often, doesn't have to be at the same version as your software, which you want to update weekly. And so this is how we allow teams to work fast. Like, HeartFlow, for example, is a team that we work with who actually uses our systems to systems architecture to release, you know, weekly. I call this out because this is really important to understand how the AI can actually or agentic features can work across these components.

So when you have a change that affects multiple subsystems, the assistant has the information of what lives in each subsystem to tell you the impact on different subsystems that your team may not be responsible for. So with that, we'll jump into this insulin delivery system. Here, we're greeted with the all items screen. This is the screen that allows you to understand all of the changes that are happening in your system. In this case, sir, we like to think of it as your change control board that really tells you what's changing across different versions.

So we have version two point o here, and we're comparing it to a previous release version, which is one point o one. And with that, we can see all of the different items that our team is working in. And by items, I mean items from, you know, six to three zero four, where we can break out many different item types that teams are working in. One thing I'll call out is that the Ketryx platform is highly configurable. These are the item types that we support out of the box that we believe are the most lean way to adhere to rigorous standards or standards like sixty two three hundred four.

But, really, we map to the way the teams are already working. So we aren't a disruptive tool. We configure our platform to match the item types, the v model, the processes, like your change impact processes that you're following, and help you enforce that in this in the tools and that your teams are working in. So we can see all the different item types, like requirements that are acting as risk controls, software items, test cases. You can see the different states that these items are in, enforce certain approval groups, like you want different groups to approve requirements or test cases.

And then you can also see the state that they're in. So some that are open or resolved or in a closed state that have maybe had their part eleven approvals, and so now we've controlled them, and they're ready to go into our documentation. I had mentioned that this is your change control board, and what I meant by that is we can see all the different changes that are happening across versions. This is showing us that we've made a change to an item with this chevron icon here. This means we've added some items in this two point o version, so we very clearly understand the addition of items.

Now this is all important because this is the data that our agentic features will use to run across. We will build this data that is rooted in the tools that your teams are working in to provide a limited context window for the agents. So we have the context from items that live in Git, from the items that live in Jira. We know the approval states. We know what's changed.

We can now understand traceability. So with Ketryx, you can view end to end traceability and see how all of these items contribute to that traceability. So we have requirements here in the user requirements column. We have product requirements. Here's that risk or the requirement that's acting as a risk control, and you can see that the risk that it's related to is traced directly to it.

You can also see the design output and the verification and validation test columns where teams are working in. And so maybe that's looking at or fulfilling this traceability with items that live in Jira or items that live directly in the source code. In this case, we have some automated tests that live directly in the source code as well that can be reported through your CICD pipeline to report your automated test execution results. This is real time traceability that teams are using to visualize as they're working. So with this bead icon, I can understand upstream and downstream traceability of impacted items.

So I barely very clearly understand that if I change this item here, what are the upstream and downstream items that are impacted? You can also filter using these controls at the top. This one here says we have ninety three percent of our design outputs covered by tests. And so you can click on this, and it will filter the traceability for you to show you exactly where you have that gap in traceability. This is how teams visualize gaps in traceability very early in their process.

As your teams are working, as you're developing new features, you're seeing this traceability being built in real time. We can also provide different traceability views so you can visualize this for what's relevant to your teams. Like I mentioned on the all item screen, this platform is highly configurable. What we're seeing here is a five column traceability view, but we work with teams who have eight columns, you know, many different levels of requirements that they're working in, and many different components. So if you think of that systems, the systems architecture I referenced earlier, you can imagine that contributing to this end to end traceability where you have multiple requirements across components filling fulfilling a top level user requirement.

And so then you can understand just logically here what's impacting what, what is gonna be affected from upstream or downstream perspective. Now this is all traceability being built by your teams and the tools they're working in. We can jump directly into this item here and see that this is that requirement being implemented directly in Jira for the the teams that are working in Jira. And so you may be thinking, well, Jira isn't really a great tool for requirements management, but Ketryx actually adds on a few different things to Jira to make it a tool for teams to do requirements management in. And that's an approvals widget, which you can see down here below, as well as traceability.

So we embed local traceability to help teams understand change. So what we're working up to here is we're understanding how Ketryx connects to the tools that teams work in, how we are building this knowledge graph, which is your traceability across the different tools and teams that, and teams that are working in those tools, how we can enforce different checks in those tools, allow you to do approvals, and enforce certain processes as well. Along with some of these checks that we automate, we also have ability to understand if an item is changed, what would be the downstream impact to that? So here's a control that's telling us that this item here, so this requirement in Jira, that the design input has changed, and you need to reassess if this item needs to be updated. And if so, you need to move it to an editable state because right now it's closed.

So you need to remove it or move it through that approval workflow and then identify what changes need to be made. This is how you can automate that change impact from items that are impacted in your system across tools. You can also view those changes directly here. So let's say I'm working on this item. I notice this flag here.

Now I want to view those changes and see what's changed. In this case, it was some red lines and green lines, but we're not concerned. So we can dismiss those changes here with an exact rationale for why we want to dismiss this reverification flag. I'm just gonna put dismiss here, and we can see now that that flag is gone. This is just one way to understand changes, in the tools that your teams are working in like Jira.

So now that we have this real time traceability being built, like I mentioned, this serves as your knowledge graph. What we've been working up to is making sure that we understand what is the underlying data that our assistant can work across and how we can use this to limit the context of our system. So now is a great time to start interacting with our wand or our assistant here. In the top right hand corner is that assistant that allows you to or that serves as your copilot and understands the knowledge in your system. So it understands this traceability that we're viewing here of what are the upstream and downstream items and what would be or what is, you know, logically related.

So what is the parent and doesn't exist? It can help you understand that by saying, you know, show me all requirements for the insulin dose calculation. It can sort and bring me that information right away. This is maybe one way to resolve some of the issues I faced earlier in my career where I didn't really have that flexibility to semantically search or where I had an AI feature built into the tool that I'm working in that understands the context of my system. So not just keyword searches, but it understands relations as well or similar context.

You know? Where am I implementing a requirement that is similar to daily insulin dose calculation that maybe I wanna be made aware of as I'm trying to make decisions about some changes that I want to implement in my system. So we understand the this knowledge graph that our traceability is being built as and how it reflects or how it contributes to our AI. And I think one other key point about this to mention is that previously, this traceability is something that teams are generating as a report at the end of their release. It's just an artifact that you have to create to prove that we have traceability and we have no gaps.

But with Ketryx, it actually becomes a very powerful knowledge tool. It is something that gives you confidence in the quality of the product you're building and allows the AI to reason across it. This is what Ketryx was built off of. It's that neurosymbolic reasoning that provides you with the logic, which is this traceability graph, and then the reasoning, which is the AI running on top of it to help you make decisions outside of that traceability. So whenever your traceability is incorrect, you may be wondering, well, how will my AI perform if I have traceability that is maybe doesn't make sense if I have incorrectly traced items?

Well, that's where you can use the assistant to perform some type of change impact analysis to help you contextualize, does this make sense in my system? And so now I'm actually going to perform an impact assessment on this change that I'm considering making. Now I'm probably taking the workflow of someone who is interested in making this change, and I wanna make a change request for or a change order. You could also take this from the angle of, we already have a bug that's been found in our system, and now we need to create a formal change order for that as well. Similar to how I'm using the assistant, you can ask the prompt or you can prompt the assistant with that request, whether it starts from a change that I'm creating or it starts from a bug or maybe it starts from a failed test execution during VNB.

All of these are different ways or areas of which you may introduce a change in your system as well as one that's found during post market. You can interact with this assistant to help you understand very quickly what is going to be impacted. Now before we go through this assistant, I just wanna touch on one more thing that that that provides context to this assistant. We talked about the knowledge graph or the traceability being your knowledge graph, but it's also the documents that you provide it in your EDMS. So you can upload QMS documents, plans, policies, procedures, your development plan, your change order plan, all in this documents module as an input to the assistant.

And I have one ready to go here that I've prompted for you to review. This is the change impact analysis for this insulin delivery system. This is just one that I've created with deep knowledge or deep research that is very general that says, hey. Perform this change impact analysis following these steps here, understand the change, assess it, and then determine impact. What I've done with the assistant is I said, let's take a very shortened version of this with maybe a few bullet points just so I can get my teams very quickly started on understanding change impact.

What we then do is we take a very complex document like this and feed it to one of our agents down here. So we'll first review this workflow of the assistant, which is something you interact with on a you know, maybe you're asking it a one off question. And then what we do is we work with teams like that use case or the case study that Megan walked us through earlier to codify it into an agentic workflow, which we'll see next. So we have the knowledge graph from all the data that lives in different systems. We have the knowledge from different documents that we can provide it, and this is how we really get that Goldilocks of context for our assistant.

Now that we have that limited context window, we can perform a change impact analysis rooted in the context of our system, and, also, it can be rooted in standards that you're complying to, whether that's six two three zero four or six two zero two six two two zero two automotive or one four nine seven one. You can prompt it with the standards that you're looking for and would like the assistant to adhere to. So I'll expand this a little bit further here so we can all review it together. But what we've done is it's provided, again, those three bullet points to walk through for an impacted assessment. What requirements are gonna be updated that are directly affected?

That's what I told at first. I just wanna know the really important ones to get started. And here, it's listing what these items are doing and then what test cases require updates. Now, again, if we think about that systems to systems architecture, this could span across many different subsystems or components. This helps teams to understand if I make a change here, how does this impact another team that's working on work somewhere else?

It's also helpful when you're looking at risks as well. So in this case, I am interested in adhering to standards like one four nine seven one. And so under doing a risk assessment is very important for every change I make in my system. And that could be particularly helpful when you have risks or harms that are represented across or reused across different components? If one team changes the FMEA, what are the impacted requirements that are acting as a risk control or maybe introduce that risk?

This is one way to understand that impact analysis as well, and we can configure the assistant or agentic workload to do that. So here, we're seeing the safety and risk impact assessment as well and the potential harms that adhere to one four nine seven one and what we can do next. So what are the recommended actions? In this case, I was saying, hey. I'm just curious to know what this change would do.

Now I'm gonna say create the change request. Now many teams call this different things, like an order form. But in Ketryx, we call this a change request, which contains all of the information that I need to properly make a change in my process. You can then take all of these change requests and put them into a change order form or a summary that you provide as a document. So all of this can be output into a document that you then can provide to auditors or for your teams that are working on it.

So what we're seeing it do here is think through all the different items in your system. It's proposing some changes, and a key component here is that we're also keeping the human in the loop. This is making sure that we agree with the description. We agree with the reason for change. And then what it does, it prompts me to review this item with with the human in the loop where you can make edits to the item.

So here, it's proposed a description. Maybe we'll say this is introduced in version two point o that we're working on. Actually. Yep. And reason for change as well as the affected items.

So not only did this understand the context, but it went ahead and associated it with those related items. And so once we create this item, what this will do is it'll actually push these changes to Jira where our team is doing this work. And so I can click into this item. Let me close the screen here. And now we're viewing the item that we just created together.

That change request that lives in Jira, because, again, this team's doing work in Jira. Here are the description impact of change, and here is that traceability we just created together. I think with AI tools where you're trying to bring AI into your workflow, a lot of this can still feel pretty fragmented because you're jumping across tools and context lives and different tools across your entire product life cycle. But with Ketryx, because we connect to those tools that teams are working in, we have the full knowledge of your life cycle. We understand the relationships between items, and then we can also go ahead and create these items for you as well.

So here we went ahead and created risks in Jira items, and you can see how this item looks in Jira as well. Push to Jira automatically with that approval, and here's that traceability widget that we're seeing directly here. Now this was the workflow for using the assistant to understand change impact, but we can also codify this into an agent. And so an agent is more of a a task that you wanted to run, every, maybe, day, weekly, and you can create those validated agents to do that work for you. You can create a custom agent, or you can use one of our validated agents that we have out of the box.

In this case, for the team that we referenced earlier, they actually created a change impact analysis agent with our help to look over the data in their system to refine a prompt here, which was that prompt that I showed you earlier in the Word doc, to feed it how we want it to go through a change impact. And let's take a look at one of those results together. So here's a table with all of the information that goes into a change impact. What we saw in the Assistant was a condensed version, but I'm sure everyone on the call here knows that there's a lot that goes into these change impact analysis. And so this is more representative of what that actually looks like when you're going to create a major change, like what we're seeing here.

And so what we've done is we've listed all the items that are impacted, and I sorted to show you the ones that are yes, but also the ones that have no here. And the reason for the no is because maybe you want rationale for that. Maybe you want to know why did the assistant say no to this and to confirm, do I agree with that? And so we provide you with that response here. And we could take a look at those items that require impact, and then we also have the rationale listed and the suggested improvements.

You can also take this document or this agent response and download it in an Excel if you so choose, to review some of these suggestions. Or, of course, you can review the suggestion, and it'll take you directly to the item to review it and make those changes. So this is just another way that we work with teams to, create agentic models that run and do these change impact assessments. And this can be done, you know, daily, weekly, every time that you have a new change that comes to your system. Once we identify what the prompt is, we then can create a benchmark.

So we take the requirements, the changes that you're already doing today to make sure that these agent results are great. We give you quantitative evidence or quantitative confidence that these results are actually improving you, getting giving you seventy percent faster time in that change impact assessment, as well as giving you defensible proof as to everything the agent is suggesting for the changes that you should make in your system. And so with this table, this is one of the ways that teams are using our change of fact assessment to automate the workflows and that manual change impact that they're doing today. One of the things that I was so impressed by when I first learned about Ketryx is how intelligent the AI agent and assistant are. I previously have tried to complete change impact assessments using general LLMs and found that it actually created more work for me than help.

And I have been so blown away by how constructive the feedback is on from both the assistant and the agent, and the ability to actually take action from it. It's really been a game changer for some of our customers, and I wish that I had it when I was operating as a systems engineer. Exactly. It it really does speed up the way that teams are working to have all of this in the platform, bringing that information together to help you understand, you know, where is something missing. I think today, are are pretty good or have processes in place that help them understand, does it exist?

Yes or no? But does the relationship actually make sense when you're thinking about scalability? That becomes a very compounding issue that that often results in manual work without having an AI tool to run across all of this data. And one more thing I just wanted to highlight, whenever you're taking a look at any of these items here that require changes, we have that traceability, widget that's built into each record. But down below for each record so, again, this is an item I think this was a software requirement that the agent said requires an update to.

I can now understand all of the documents that contain this record that would require updating. And so, typically, I've I've seen teams that, keep a manual Excel that says, go update this document if you make a change to these set of requirements. But because we have you at an item based workflow and keeping track of this of the metadata instead of the document level, we automatically know what documents would require an update if I were to take this agent suggestion and make this update. And none of that even is manual. This is more of a visibility.

On the releases screen, if we're going to our two point o version, we can see all of the documents that would need to be updated. And so because we're managing that underlying metadata, all of the different items, we then pull that information and put it into a templated document for you with our robust templating language that is custom for all of the teams that we work with. And when it comes to updating it, it's as simple as clicking this generate documents button to pull in the most recent version of all of those items. So if we did make that change to the requirement that required updates to those documents, we now here can quickly update our documents as quickly as we update our code. Baylor, I'm seeing a question in the chat around how risk management connects to the AI assistant change management.

Would you mind walking through that? Yeah. Absolutely. So because Ketryx also has its own risk management module, we can manage the different risks in our system. Now teams manage risks in many different tools today, whether that's Jira, Polarion, or maybe even Excel.

We can bring you to that item based workflow and allow you to manage risks here in Ketryx. What we're viewing is that risk table here where teams can trace two risk control measures directly in their directly where these items live. And so if I were to change an item to that requirement that acts as a risk control, we immediately would know based on the relations in here. And also from response, when we interacted and did the CIA with the assistant, it told us the impacted risks. So it looked at the risks that we already had here in the system today to say, hey.

I think this change will impact this risk that you've identified already. Or maybe it's going to introduce a new risk that you don't have, and we suggest you create it. All of that can be automated with the assistant in that change response to help you bring in your risk activities into that assessment. And then finally, you can view all of that as well in your risk or in your traceability module. So let's say we did actually create that actually, here it is.

Here is that requirement that we created together or the CR, excuse me, for the lowering the ML confidence threshold. We created the CR together. We created the requirement that was impacted, and we created the risk traceability. So we went ahead and created this entire row of traceability using the assistant and understanding exactly as well what risks would be impacted or need to be created based on that change. Thanks, Bailey.

It's, like, so straightforward or it's so simple to see it all laid out like this and get that visual map of how everything relates. Right. Yeah. And then I saw one more question about being able to trace to QMS docs. We do have that capability as well where you can store your QMS documents here, where you can store other relevant information.

Your QMS documents can be stored. But we also have the ability to create documents as items. And so, for example, this team actually does have, you know, their SOPs as items where then you can then build traceability to the items as well similar to how you would build traceability to other items. I'm seeing another question here about when we're creating a new custom agent, maybe, like, the AI diagnostics company change impact assessment agent, How do we help teams evaluate and verify the agent results? Bailey, do you wanna speak about what that experience has looked like for you and your team?

Yeah. Absolutely. So that is a a really important part when you're using agentic workflow is being able to create benchmarks to assess, you know, how this agent performed with real data. And so with that team that we worked with, we tick we took their real live data. So we took their requirements, the common changes, and we compared it to what they found prior to Catchrix.

And then you ran the assistant with the same requirements, so same data, and then determined how it performed. Did it catch all of the requirements that we knew were impacted, or did it find more that we didn't think of? And so we worked to train and refine the prompt to make sure that it was comprehensive of the information we are looking for. And then, of course, we ran those results to to find the proof that it actually did find, you know, like, eighty percent of the requirements that were affected and even more that they that weren't caught by the human eye. So we will work with your teams to create that benchmark for testing.

Alright. And then I think I saw one more question about oh, okay. That was it. I'm looking at the chat. Wonderful.

Well, thank you all so much for for joining us here today. This is a topic that Megan and I are both very passionate about. It is something I talk with teams about daily when I'm introducing the Ketryx platform. I think everyone is interested to know right now, you know, how can we use AI, but how can we do it in a meaningful way that actually makes changes, that makes recommendations that are meaningful for our product development? And I'm really proud to to work with Ketryx and have a tool like this that helps me day to day demonstrate that.

Absolutely. Thank you all for making the time to be here. As mentioned before, we'll be sending out the webinar recording after this concludes, so you'll have that at hand. There also will be a survey for you to complete. And if you're interested, we are having a webinar on Tuesday, March seventeenth, talking about how we worked with Flow Health to build regulatory readiness fast.

Our partnership with Flow allowed them to go from unregulated to regulated in ninety days supporting their first FDA submission. That's a really exciting story, and if I would love to see you there if you wanna join. Thank you again, and have a lovely day.