---
title: "ADO for IEC 62304 and AI - Developing FDA-Compliant Software in Azure DevOps"
type: webinar-transcript
publisher: Ketryx
source: "https://fast.wistia.net/embed/iframe/rkya2vtu1j"
content: auto-caption transcript, proper-noun corrected
---

# ADO for IEC 62304 and AI - Developing FDA-Compliant Software in Azure DevOps

*Ketryx webinar — transcript of the recorded session.*

[▶ Watch the recording](https://fast.wistia.net/embed/iframe/rkya2vtu1j)

---

welcome everyone to this today's webinar on Azure DevOps for IEC six two three zero four in AI. I'm your host, Gabriel Pasquale, and I'm joined here by Carlton O'Neil, one of our product managers. We'll give a little bit more background, on ourselves in a moment. But first, let's looks let's look at the the poll results. So we'll share those over.

And let's see. Carlton, it looks like the the control oh, there we go. That is working now. So as I as I mentioned before, there will be a recording and slides sent to everyone after the webinar. Please put any questions you have in the q and a.

We have our colleagues on the line that are monitoring for questions, and we'll surface to them as we present. And then finally, whenever you leave the webinar, we have a feedback survey. We'd really appreciate you providing any input both on this content or content you'd like to see in the future. We wanna understand what you are most interested in learning about within the space of regulated software and product development as well as around specifically what CapTrix has to offer. So just sharing the poll results on how your team uses ADO today, I see a big, chunk around requirements management.

So we'll talk we'll talk a fair amount about that today and in particular in the demo. And then on the CICD pipeline management, that's a very common use case that we see in a tool like Azure DevOps that has everything integrated together, all the way from requirements management through automated testing and, ultimately, the artifacts that you you push to your customers. And then we see some significant adoption around ADO. And a a challenge around, I would say, we have ADO but haven't fully adopted it. And I think that can be a a significant challenge for regulated teams because there are ways in which we can use ADO kind of out of the box.

But when it comes to other activities like requirements management and and producing documentation, we could see some challenges there. Great. So quick introduction on on Ketryx. Ketryx is a connected life cycle management system. We like to say that Ketryx enforces the processes you already have and the tools that you're already using while automatically generating evidence of compliance to ship products faster.

Today, we'll be focused on how teams can leverage Azure DevOps while getting all the added capabilities of part eleven compliant approvals, end to end traceability and risk management, as well as ultimately producing evidence for a regulator or an auditor. Now for some quick introductions, Carlton, I'll have you start out with your intro. Yeah. So glad to be here. I came to Ketryx from Amgen, was in a variety of roles with Amgen ending up in the process development function.

And the entire time I was at Amgen, I was both executing within the quality management system and executing on the quality management system in the form of, managing capas and deviations. So very familiar with the, compliance side of things. Joined Ketryx, worked in client operations, and I've had the privilege to work with, a ton of different, software companies, help them, speed up their releases, by using Ketryx. And now I'm on the product management side, ensuring Ketryx, goes in the direction that our customers need it to based on all my experience working with, medical device manufacturers primarily. Thanks, Carlton.

And I'll and I'll just add and and maybe, you know, toot your own horn there in saying that everyone that you've worked with from kind of Fortune one hundred companies and implementing, you know, very complex products, robotic surgical systems, advanced software and AI, all the way down to kind of the fastest moving startups, that are looking for and and looking for a solution to use modern development tools. So, hey, everyone. I'm Gabriel Pasquale. I previously I I started my career at the MITRE Corporation, actually, as an embedded system security engineer focused on cyber physical systems and led the innovation portfolio at MITRE, which had a significant focus around health care. I then also spent some time at Amgen where I ran into a lot of the challenges of validated AI development, and, you know, what it takes to to take an algorithm and operationalize it, validate it for for its intended use.

Finally, I'm now at Ketryx, where I'm helping teams get over these challenges and then leverage modern software development tools to execute a software and AI life cycle, under the constraints of standards like six two three zero four, and GAMP. So today, this is our agenda. We're first gonna go over talk a little bit about modern development tooling. We'll go over some of the key requirements for six two three zero four compliant development, you know, and then specifically in the context of six two three zero four with an ADO. As a part of this review, we'll go through a live instance of Caprix integrated into Azure DevOps.

And then finally, we'll close it out with a set of questions that that you can take as you are considering adopting, a total product life cycle approach, but particularly in the cons in in in the context of the tooling you're selecting or the tools that you already have today. So we'd like to start most of our presentations out, with this particular slide that talks about the complexity of software and products in the regulated space growing exponentially over time, but that the productivity of our teams are not, increasing at that same rate. And our our belief or our thesis is this is largely around the challenges of tools and process working together in concert to ensure that we can not only innovate safely, but that we can deliver software and solutions quickly to the market. So we can contextualize this problem within the existing tools that many software developers and product teams love. Tools like Azure DevOps are built to help manage the complexity of building systems, AI systems, web systems.

And there are a few core areas that they help with. It's largely around project management and collaboration, enabling your team to follow agile processes, providing a set of tools that your team is familiar with so it's easy to onboard new staff. And then finally, that they're customizable to your particular process or workflow, depending on the type of product, in the in the market you are deploying that application into. Now, really, there's been an evolution of developer tools over time, And this plays into kind of the larger thesis that we have around not having the best support for regulated teams when they go to use these developer tools. Developer tools have a foundation in safety critical industries.

Many of the first tools were developed for aerospace, defense, automotive, and telecom early on in the the eighties and nineties. Now over time as the agile methodology evolved and tools like Jira and Git and Azure DevOps were developed, the need to support these agile processes grew, and therefore these tools emerged. Following agile development, we have the emergence of DevOps and cloud based applications. And so tools for managing the increased complexity of web scale applications started to emerge. And then finally, in the last fifteen or so years, we've seen the emergence of data driven applications and AI and an additional set of development tools that are necessary for not only managing the life cycle of the software, but also the life cycle of the data and the models that are trained on that data.

And ultimately, where we are today in the last two or three years with large language models, ChatGPT, Anthropic, all of these products that leverage kind of agentic, workflows have added an even an additional set of tools on top of that. But what has largely been left out from this evolution is how to take these tools and apply them within a regulatory context where we have the ability to, follow a specific quality management system and ultimately collect the necessary evidence, which is both documents as well as proof that we followed a particular process that show that we're able to leverage these tools and ultimately produce those artifacts that we need for regulatory, agencies and audits. And this is a story, a broader story of how many tools are necessary to ship software today. Across each phase of the development life cycle from build to release to operating to back to planning, We need many tools, and we need to coordinate not only the the development of the product, but also the development of the documentation deliverables. So today, we're gonna talk about developing IEC six two three zero four software within Azure DevOps.

And as you know, there are a number of different stages across IEC six two three zero four that require us to do a set of planning, artifact development, testing, and so on. Now this is a bit of an art eye chart. It'll be available when we when we send over the slides, but I think it does a good job of describing each of the activities that we need to follow as we go through six two three zero four, what audits or reviews need to occur, and, ultimately, what evidence needs to be produced at each phase of the development life cycle. Now with respect to Azure DevOps and how today it supports IEC six two three zero four compliance, there are some challenge areas that we'll cover today. And we'll break that down across four separate sections of IEC sixty two thousand three hundred and four, section five, six, seven, and eight.

We'll focus specifically on the challenges of traceability, signatures within Azure DevOps, and then finally the challenges around ultimately validating all of the automation that you've built into your development tools so that you can use them for evidence collection and generation. So now we'll start one more poll that will focus on the challenges of activities that you are currently executing in Azure DevOps today. And I think, you know, just from our work with folks that are using Azure DevOps as I see some of these results come in, you know, it's it's oftentimes, around traceability, oftentimes around how to, you know, properly execute change management activities, kind of understanding the the the scope of changes that are going into a release compared to the previous released baseline. And, ultimately, that feeds down into document generation, approvals, and, testing. So we'll give it a few more moments on this poll, and then we'll we'll share the results.

Great. So we'll go ahead and end that poll and and share the results. And and just looking at it, I think, you know, ultimately, document generation is a is a big component. And, you know, Carlton will definitely focus on on on generating documents. I think that's one thing that we we focus on.

But, ultimately, how do you how do you produce all of the the evidence that's necessary, upstream? So when it comes to ADO and and these challenges, a lot of these are are limitations in some way by design. ADO doesn't necessarily help you maintain immutable records or do electronic signatures. Oftentimes, have to find plugins from the Azure DevOps marketplace in order to execute signatures. Enforcing control over the total product life cycle just not necessarily meant for helping you provide, a management solution from product realization all the way through post market surveillance activities.

Third is challenging challenges around tracing across other systems. This challenge of needing so many IT tools and software to execute the full product development life cycle really frustrates, the documentation process if you need to trace, let's say, from a system where you manage requirements to another system where you have your software bill of materials. Fourth is focused around this NPS validation, which we'll talk about in some depth, and making sure that any automation that we build into our our product or our quality system, is in in fact validated. And then finally, on the ADO side, specifically, challenges around relationships and configurability. I think we'll we'll talk in some depth around the types of relationships that you can build natively within ADO and some of the constraints that you might be bumping up against, as you work with that.

So, ultimately, I'm gonna hand it off here to to Carlton, but what we're gonna talk about today within the demo and particularly aligned with six two three zero four is how do we bring the best things in Azure DevOps relevant to simplifying product management, it being a tool that developers already use and already familiar with, with items like part eleven compliant audit trails, submission ready evidence, and end to end traceability to ultimately allow you to achieve validated weekly releases. So with that, I'll I'll hand it over to Carlton who will start to bring us through six two three zero four and the demonstration environment. Awesome. Thank you, Gabriel, and thank you for a very, well managed introduction. So as Gabriel said, we're gonna go through the different, kind of top level sections of six two three zero four.

Each section that we're gonna focus on will have a slide much like this. The top bullet points are kind of explicitly spelling out what the section contains. In the middle, we will give kind of solutions, for ADO, to meet the requirements that exist in, the in whatever section we're looking at, and then talk about some challenges. And some of the stuff in the middle will address the challenges below. And then, of course, we'll show you how we approach these challenges.

So the first one is, kind of the heftiest section, which is the software development process. And this has everything from the top level requirements and user needs down through the system and software requirements into the detailed design, and then across to our testing. It's kind of the v model section, of six two three zero four. Within ADO, as I'm sure many of you do based on the pull response, you can manage these configuration items, using work items, in ADO. And you can use the out of the box work items, which have some overlap with the concepts that, are contained within six two three zero four.

You can, of course, create your own work items. You can use out of the box fields. You can use custom fields, state transitions, within ADO to kind of build your v model. Azure DevOps also has linking to manage traceability from the, you know, left side of the v model across to the right as well as up and down. And then, of course, there's some testing, functionality dashboards, and kind of the normal project, management tools.

And then you can have, electronic signature plug ins. With this, you can kind of cobble together a a workable solution, but, certainly, there are challenges that that will be presented, just by using plug ins or or native ADO functionality. One, which is the biggest one, is most of those systems aren't validated as nonproduct software to run a six two three zero four compliant process. That obviously, puts some burden of of nonproduct software validation on the team. So instead of building the product, building and validating your product, you find yourself, spending time validating the product you need to build your product, which, of course, is, maybe unnecessary overhead for for the product teams.

And that also has the, the challenge of, you know, if you're gonna have electronic signatures and immutable records in ADO, then, of course, you need to validate it. If you're gonna make traceability in ADO, you need to validate the links, and, you know, the robustness of those links. And then, also, there's a challenge of, these are your design controls. Your your design controls need to version with the actual software and managing the versions of the design controls, as work items alongside the versioning of your software can be challenging. So I will show you how we approach, this very large section of six two three zero four.

And please put put questions in the chat. The demonstration here is only gonna get better, the more questions go into the, the q and a section. And throughout as as you see, CapTrix, as you see kind of how we consider these challenges in ADO and address them, please feel free to ask, throughout. Okay. Great.

So this is the Ketryx environment, that I'm gonna show you today. It's connected to a variety of systems, ADO, GitHub, Valerion, Jira. Of course, we're gonna focus on the ADO connection here, and that connection is instantiated in this, software subsystem one. And you can see I have a bunch of items here. Many of the items that are that are considered by that section five of sixty three zero four, requirements, test cases, software items, which, has the detailed design.

And you can see for almost all of these items in the source column, we have an ADO logo and ADO's work item number for that configuration item. And then towards the bottom, we do have get based items. I'll touch on that briefly in a bit, but let's get the basics down first. If I click on one of the links, then I go straight to ADL, And, hopefully, I land on a very familiar screen to, everyone in this webinar who uses ADL. It's just a normal, work item screen.

And this item specifically is a requirement. It's a custom, work item type that I'm using. Of course, it could just be an out of the box work item type. Ketryx just needs to know, where you store your requirements or which work items, make up your requirements. So this, work item has some fields, of course.

It has some sections, that are have has specific fields. And Ketryx really does three things, to ADO to address the challenges that we talked about on the slide. The first two, I'll talk about, here in the demonstration. I'll jump back to the third one, which is traceability, which is a big thing that deserves its own slide. So the first two are, one, we inject approvals directly into ADL, and Ketryx is a validated solution.

So this, approval, behavior, as well as the rest of the behavior I'm gonna show you, comes with the validation package, which hopefully reduces the burden of nonproduct software validation for the team. So you can see that this Ketryx approval widget says this requirement is currently open. Transition it to in progress once work starts. That's one of the key, kind of helpful things Ketryx does. This is very, very easy to configure, so we can inject work instructions directly into, ADO from your quality management system.

So if you have some workflow that has more states, that may be complicated for the team and the team needs to know know what to do next, it's very, very easy to just tell the user, you know, this is exactly what you need to do next as you move this item towards approval. Gabriel, do you wanna take that question live, or do you want me to? I I think I mean, we can we can let's take it now. I'd love to hear your perspective on it too. So the question is, TIR forty five clearly states that user stories are at a level higher than requirements and thus would not be sufficient as SRSs.

Can you discuss the ADL capability to trace from user stories to SRSs? Then to define definition of done for user stories aligned with six two three zero four requirements for each story. So this will make up this will make a lot more sense when we get to the the traceability screen, just for a visual. But maybe, Carlton, you know, you wanna give an answer to this one? Yeah.

Definitely. I think, you know, certainly, the user stories are at a are at a higher level than system requirements, and and they get read in as such to Ketryx as, you know, kind of the far left of the traceability matrix, if you wanna talk about it like that, or the top of the v model. And those user stories and like Gabriel said, it's gonna really come, full circle as we get to traceability, but those user stories have child relationships downward to system requirements. Those system requirements are implemented or or fulfilled by, some aspect of the software architecture or detailed design, and then we can test across and and kind of split it up based on what the test case is traced to. So if we have a test traced to a user story, then that user story is the far left of the traceability matrix and the test is the far right.

Or if you wanna think about it in a v model, it's the top, top row. And then we can kind of flow down through the work items and through the traceability to system requirements to, how they're fulfilled and then back up the other side, you know, verifying and validating as we go to, of course, in the end, build the right thing and build the thing in the right way. So, Gabriel, I don't know if you have anything to add there, but certainly, something, that I'll continue kind of chasing as we as we talk about traceability as well. I would just say call calling back to, you know, ADL is is quite a flexible tool. So creating, item types for requirements and stories and and definition of done is is all relatively easy to do from a configuration perspective.

I think a lot of challenges come into visibility and understanding the system as a whole as you develop it. So I think Carlton will show this Ketryx traceability tab in a moment. But one of the challenges for teams that that are in the tool is getting that context. Hey. I'm working on an SRS.

What are the the upstream store stories that relate to this SRS? So rather than having to go through and and query and filter, CapTrix is here to help you get visibility into that structure, both within the the global trace matrix, which we'll show in a moment, and then this local trace matrix that Carlton will show shortly. That's awesome. And right before we get there, just to make sure we have, the basics down, I'm gonna update this, item just to kind of show you the the first two things that Ketryx, provides in ADO. I'm gonna update the title.

I'm just gonna say change, and that is all I wanna do. So I'm gonna move it to resolved, and I can save that as normal in ADO. This item in ADO had just had an update. That will trigger an update in Ketryx. So if I go back to Ketryx and I look at this same item, and let me make my column a little bigger, we can see the change is already in here.

That's the the update I just made to the title, and Ketryx has it in a resolved state. Now that it's entered a resolved state, you can see the your approval column has entered pending. That means I'm in a group that needs to approve the requirement. And if we refresh the ADO page, we will see that Ketryx has will update its approval widget. So before it said this requirement is open, You know, please do this and that to move towards an approvable state.

And now it's saying this requirement has been completed. It's it's signaling that it's in the resolved state. Ketryx knows that that's the, ready for signature state or ready for review state. And now it's instructing me, you know, please review it and approve it. And, of course, I can review all the information here, and you can see I'm a part of every group so that we don't have to have four people here to, to sign it off.

But, of course, you could imagine, you know, kind of one person from each group needing to approve this. And I can approve the item. We can have a multifactor authentication signature pop up, as a pop over here with, you know, face ID or touch ID or one password or the operating system password, variety of methods if you wanna collect multifactor authentication. But, eventually, I click, approve, and I'm in every group. And you can see it says now yes for every group that needs to approve it, And the widget has updated to this requirement is closed in a controlled state.

And Ketryx moved the ticket, the work item ticket to closed. So I didn't change it to close. I had it resolved. Moving from resolved to closed, was gated by the approval. Once I gave that approval, now it's enclosed.

So any item I have here in the project that is enclosed, I know has been approved as well in a part eleven compliant manner. That's the first thing is injecting this approval straight into the work item itself. The second thing is every single change that you make in ADO will be tracked in Ketryx. So we get an audit trail of all of the changes. And you can see that it's not just the state changes, it's also the content changes.

You can see my change here in the title is reflected on this revision number five. If I go back to the revision number four, change does not appear, and we do have diff capabilities where if I look at number five and diff back to number four, it says, here's where the change is. It's in the title, and you can see the redlining there indicating that I added that word, or a hyphen, and that word changed to the title. Cool. And this is how all of the items that are in, this project are managed, with the exception of the get based items, which, again, I will I will touch on later.

But all these different work items with different types, different fields, different workflows, are being read into Ketryx. All of their content is being read into Ketryx. Their state is being read into Ketryx, and then Ketryx is providing that approval right in the work item in ADO. And then also with every single change that's happening in ADO, restoring a compliant audit trail. Okay.

Let's go back to the slides, just talk about traceability, and then we'll jump into ADO and look at that third thing that Ketryx provides in ADO, or directly in ADO, which is that Ketryx traceability tab that Gabriel alluded to. So traceability is, king and kind of probably the biggest activity, that happens in a six two three zero four, compliant, life cycle, and biggest meaning, what you spend the most amount of time on, especially when you have gaps or it goes wrong and you need to kind of piece back together. So there's lots of different types of traceability. Of course, there's the traceability that exists, in the v model, user stories down to system requirements, down to software requirements, down to detailed design, and then across to testing. And, of course, you might have a different v model based on your QMS and your risk classification.

That's kind of just an example. But then we also have traceability out kind of outside of the v model or in a third dimension to the v model. And the big one there is, of course, to risks. Items in the system, whether that's a requirement or detailed design can introduce risk, and then we need to, you know, score the risk, control the risk, and then do some residual scoring to prove that the risk has been controlled to an adequate manner or provide some benefit risk analysis if it has not. So within ADO, and I mentioned this earlier, you can use work item links to maintain that traceability between the different work items and then use the built in boards or queries to try and, assemble the traceability matrix based on those links.

And I think the challenge we see typically there with, the traceability is that when you have to generate it, you know, using export or Power BI reports or something, it's not necessarily real time. So if I export the traceability matrix and I realize I have a problem, I can't just update that Excel sheet and then have my traceability be covered. Now I need to go back to ADO. I need to go find a work item, that's missing a link. I need to make a link to the proper other work item, and then I need to rerun the export.

And this just creates inefficiency from going back and forth from the export back to the system. And so what we do is we take the links that exist, between the work items. And in ADO, we give a local traceability matrix. You know, here's what this work item is connected to, and that's a work item of any type, connected to any other work item of any other type. That's as many connections as you want.

A requirement could be covered by forty different test cases. Forty requirements could be covered by one test case. It's a kind of end to end relationship. And then we provide also a global traceability matrix, which is, instead of exporting the work items and their links and then having to go back into the system to update the work items when I notice there's missing links in the export, We give a view of all the work items, all their links, and when you have a breakage, we show it to you, and then you can go straight back into ADO and update, that work item. So let's have a look for this item at the Ketryx traceability tab.

And this is just kind of another tab in ADL. This will kind of automatically be injected into the environment, and it's using the work item links on our requirement that we've been focused on to build this graph. And you can see that it is, kind of building upwards and downwards. You know, as we think about design verification, I can see that this requirement has a parent requirement or could be a user story. And you can see this one is provider reviews patient data.

That's probably the title of a of a user story or a user requirement or user need. And then I have some more specific, text that exists within this requirement. And then I also have a child, which could be detailed design, could be a software requirement. And through this path here, I have kind of the full left hand side of the v. I also have the testing.

I can see that this requirement is connected to this test. I can also see what else this test is connected to if I expand. So I know, oh, this test is testing this requirement, but it's also testing these four other spec items. I can also see execution results, and I can see that one's marked with one dot o, and it has a green check. So that kind of helps us expand the the view here to not only look at the directly related work items, but also look another step away, as we're thinking about local traceability.

I also have a change request in here that affects the requirement. I'll talk more about kind of changes to the system later. But, if it's connected and it's a work item, we will see it we will see it here. These are also clickable. So as a user, if I'm, working on this requirement and I update this requirement and I wanna go update the test, I am able to click that, and it goes straight over to the other work item as you would expect.

I'll also mention here that Ketryx has some automation, which is really, really helpful, as we go through the design verification process, which is configurable. You can turn it on and off with the with the checkbox, but we are able to kind of force changes down the v model, if changes happen upstream. So if I change the parent item, but, you know, provider reviews patient data, that might be a reason why, I wanna change the system requirement. We can, move this system requirement automatically from closed back to resolved such that we need another review just so that we have the chance to, check and make sure the system requirement is still, an adequate child of the user need. And same, of course, with test cases and and any other, work items.

We also have the ability oh, sorry. Go ahead. I was gonna say, I think that's a really important one to double click on, particularly for highly distributed teams that are working on complex systems. I think we like to think that, you know, we can we can manage and understand these changes across time. But once you have a a ton of items, you have multiple subsystems that you're maintaining to develop, an AI product, you need some system in the background that can monitor this for you.

And if you do wanna get to the weekly release state, as as many of our customers have gotten to, this is is kind of a core part of of, the the solution. Yep. Definitely. Definitely. And it allows you to confidently work across traceability rows instead of down traceability columns.

Like, if we have this validated automation, in the system, then we don't necessarily need to complete all of our user needs or user stories or user requirements before we move on to system requirements because we can be confident that if a row is changing, if I'm changing one, user need, then all of the items in that row are gonna require reverification. So I could work across my rows at different rates with full confidence that, change is being assessed. Of course, specifically change is being assessed, in accordance with, the risk management procedure that is effective for your organization. I see we have a phenomenal question in the chat, which is how does Ketryx maintain traceability across multiple ADO projects that contribute to a single device? This is a great question, and I'll even expand it to say, it doesn't just need to be an ADO, project.

If if we were using other tools, GitHub, Jira, TestRail, Polarion, and I won't go down the full integration list, but we would see those items connected here in the Ketryx traceability matrix as well, which is kind of not possible, typically in ADL to connect, you know, to a to a Jira item, for example. You know, maybe the product team works in Jira, but the development team works in ADL. We, you know, often see gaps in traceability or more often manual piecing together between those Jira items and ADO items. That's not necessary here. In ADO, I'll see in the linking options the ability to link Jira tickets and vice versa from Jira to ADO.

And then the Ketryx local traceability widget will show, kind of all of the connections and then also the global traceability matrix, which is where I will take it for the what the question actually was, which is how do we maintain traceability across ADO projects, when we're managing kind of different subsystems or different components in different ADO projects. And it is a very nice segue into the global traceability matrix. So we we're looking at the local traceability earlier just for my one requirement. Now this is all of my items. These are the columns that that I've set up.

It's super easy to kind of just drag and drop columns and add filters, within Ketryx to make sure that this matches your v model. But I can see here here all my intended use or user needs or user stories, lots of different words for the top level of that v model that are fulfilled by product requirements. They're filled by software system requirements that have design outputs and then our testing, linking back this way. In this view, I can see well, first, I can change the grain. So if I wanna group by intended use, I can do that.

And now I see I have three kind of top level requirements, and, everything else is grouped according to that. I can group by product requirements. That's how we started. Software, system requirements, design outputs, and you can see the grain of the matrix is changing each time I click. I also mentioned that we can quickly see, what's uncovered.

If we look at these checks across the top, we're just checking to make sure that each item in in a row has a corresponding item according to our quality management system rules. If I click any of these buttons, then I very quickly see what needs to be changed. So instead of exporting and trying to assemble a traceability matrix to figure out if I have a problem, as we're working, again, across traceability rows, I can click a button, and I see, this requirement doesn't have a design output. Of course, I could find an existing design output and link it. I could also create a new work item, whatever I need to do, and, of course, I just click straight into ADO to fulfill my traceability.

We also have a concept of the kind of timing and the releases. Each of these items have a life cycle. They get introduced in some version. They get obsolete in some version. And as we version the product and we're about to talk about software maintenance, so this will come in very key.

We will have kind of different items showing up here as items are introduced and obsolete and as change is affected on the system by change requests and anomalies. Okay. So then to get to the the big question of how do we maintain this if we have multiple, subsystems with within different ADO projects, And kind of the key advantage there is you could version the components at different rates instead of having kind of a giant project where you export all the items and then have to fix the traceability for everything. I could use version configuration management to version just, kind of one component. And, ultimately, within Ketryx, what you can do is pull all the items from many different projects into, kind of a full traceability matrix.

And, for example, this one has, kind of more columns, and, it's coming from multiple projects and multiple systems. We have those GitHub items. But kind of these far left items are from one ADO projects. This column's from one ADO project, and then this column, subsystem requirements, is actually all of the requirements from that project that we started in. And you can see that we're able to maintain the traceability across.

So the subsystem requirement, which is in from the original project, one is incorporated here into this kind of complete traceability matrix, and then two has relationships upwards to items that exist within this top level project. Okay. Great. And then, let's go back and talk about the next section of IEC IEC sixty three zero four. Hopefully, that provided a good overview about how we can manage these work items, how we can get them, to an approval state, how we keep the audit trail, and how we look at traceability.

But then, of course, there's the aspect of of time, which I've hinted at. But, you're going to keep releasing. And kind of the key value driver is to release faster while, of course, still following the quality management system so that you don't have to do any rework, or any kind of remediation, by deviating from the process. So the question is, you know, how do we, follow the process faster? And, you know, in ADO, of course, there's there's the ability to manage complex software systems and manage the work needed for complex software systems.

And you can leverage that. Kind of a key thing that that Catcherix will provide with that audit trail is we have that change over time so we can quickly look at what items are new or changed. And as we're thinking about, you know, making a test plan or controlling risk, then we know exactly which items are kind of the most important for that version based on what's new, what's changed, what change requests are we introducing, what anomalies are we fixing, what anomalies did we discover in this version. And that's kind of the challenge here, which is item level change control as well as patching parts of the system while another release is already being developed. So to look at first item level change control, within and let's go back to our original project.

Within Ketryx, we can compare versions. So from version two point two compared to the previous release, which was two point one, I can see show me all the items that are new or maybe all the items that are new or changed. And that's really what I can focus on for this release. And especially if we have automation of that design verification process, where we're enforcing that that rows are reassessed as as upstream items change. As I approach a release, these newer changed items are really what I need to focus on maybe and probably also my risk controls, especially if they're newer change, but in general, testing risk controls as well.

You can see that we have the diff indicator showing that these two requirements and this one test case has changed, and then we have all of our new items in the system. We have change requests, anomalies, that are being introduced in this version. And then we also have some new requirements that are probably a result of those change requests and anomalies. These change requests are, of course, like everything, just another work item in ADL. They have the approval ability as well.

They have the full history, and then they have a particular version where they're going to be considered, in the design controls, especially for the change request verification report. And, also, of course, these items have traceability just like all the other items to the items that they affect. Great question in the chat. It's this question is, where does the mapping of tickets to releases happen, Ketryx or ADO? That will happen in ADO.

So just like how you would mark a fixed version or a or a version on the ADO ticket, you can do that here. We are using an introduced inversion field. You could use any version field that you like. And for point wise items, which would be like change requests and test executions, which really only apply to one version, you would just mark an introduced in version. For everything else, that will be a long lived item.

And so once you introduce a requirement to the system, the system will continue, knowing that requirement is relevant, as we go through versions. You don't have to copy the item or or anything. Ketryx will just kind of keep moving it forward until you obsolete it. And kind of the key thing that can help you really speed up the release cycle is, as long as the item isn't changed or nothing proximate to it is changed, it will maintain its approval status as we go through releases. And so kind of the key point of item level change control is I can edit just a subset of the items in the system and very, very quickly iterate.

And if I need to make a very crucial change, especially to resolve a a critical defect in a patch release, I can just change the items proximate to that anomaly, and, you know, release the same day with fully updated design controls and not, having to go back and kind of remediate the documentation after the patch release. The the patch release urgency, can be met by the design control management, which I think is is very, very rare. Okay. I'm gonna quickly take this opportunity that we're talking about change requests and anomalies, which are kind of the two different work items that are affecting design control change within this project, to highlight our agents feature. And kind of a key thing with anomalies and and change requests is are they written according to the quality management system?

Are they being risk assessed according to the quality management system? Are they being clearly defined, by whoever has filed it, whether that's some kind of customer facing team or a product team, such that the developers, know what to implement. They know which, design controls are gonna be affected. And to really, really check that, we have an LLM in the platform that has access to all of these items. It knows, how you write your, your change requests, your anomalies.

If you store your quality management system in Ketryx, it can also check the anomalies and change requests against the procedures that dictate how the anomalies and change requests need to be authored. And you can schedule this to run, at some frequency. You can also run it on demand. And what Ketryx will do is read all the anomalies and change requests that are in a particular version. And you can see I have my three items.

This one was a change request. That one's an anomaly. This one's also a change request. And Ketryx will provide some, recommendations. If we go back and look at our chain my change request, you'll see it is completely blank.

So, of course, Ketryx has some recommendations here. First, no specificity in the change description, and you can see I have the ability to, have KI suggest a change here. So I KI has access to all the design controls. It has access to the procedure. If we want a quick first draft of the change request, I can click this apply button, and you can see it has, other suggestions as well, of course, because the item is completely blank.

No specificity and change description. No measurable success criteria. No impact assessment is done. An unclear rationale. Well, it's blank.

I haven't done any of that, but, you know, those are things that are defined in, my quality management system that I have in this project that I need for every change request. And you could imagine with hundreds and hundreds of items how helpful it could be to be able to very quickly find the change request or the anomalies that are not authored according to the quality management system. And, of course, finding it now instead of later. We find it later, then we waste time. We're inefficient in fixing the bug or or implementing the change request.

If we catch it kind of at build, if we catch it afterwards, of course, we might have to cut a process deviation that might involve a kappa, which, the more you limit kind of the quality records, especially deviations in kappas in the system, the less, your cost of compliance is. Okay. So as we change the system according in accordance with, software maintenance, of course, we run into risk management. And risks are, other work items in the system that can be introduced from requirements or from detailed design. And the challenge here is that risks need to follow some scoring.

You have some, probability scoring, some severity scoring, which leads to the acceptability of the risk. And it's you know, the the work through the matrices is is nonnegotiable. If it's, critical severity and, extremely high probability, you know, that needs to be assessed as a major risk or or a critical risk even. And in ADO, of course, you can have fields that are, select fields. But what you wanna do is take the severity and probability and enforce a risk assessment on that risk.

And there is a plug in, risk priority number in ADO, which can allow you to kind of combine fields together. But, you know, again, another thing to validate, if you're gonna go that way. Of course, you could also build some custom automation in ADO, another thing to validate. And so what Ketryx does is these work items that are risks, you can just enter in the, kind of what Ketryx needs to do the calculation. So I can you can see here I have some classification field, and, of course, I also have hazard analysis fields.

And the classification fields allow me to score the risk, and then Ketryx does some calculation. And, ultimately, what I get is a final, kind of overall risk, acceptability from both the initial risk, and then, of course, we control it, and then we have residual risk scoring after that. And then these items that are risk controls are very, very important. We need to we need to test those. And so the other view we have, which looks from the other direction, is here are all the risk controls.

Here's whether the hazard analysis has been done. Here's any risks arising from the risk controls, and here are the tests. And kind of a newer thing that that we need to consider with regard to risk is and, I guess, not a newer thing, but but the agency has been very clear that we need to connect kind of security cyber risk into the fourteen nine seventy one compliant risk process. So, of course, that brings up SEWP management. And in ADO, you can define a custom work item SEWP and then try to track SEWP vulnerabilities using, a plug in like SEWP.

And then kind of the challenge you'll run into is connecting this item, into especially the vulnerability into the risk assessment process. And the way we consider this is we can scan the SBOM or you can report a CDX to, kind of populate your list of dependencies. We can find vulnerabilities or you can report vulnerabilities, and then these items can be assessed and then connected to the fourteen nine seventy one compliant risk that we are just looking at in the system. Awesome. And then kind of throughout what everything I've shown, this all comes into the documentation for the release, you know, the traceability matrix, the system requirement specification, the testing report, the test plan, the risk management file, the risk matrix, all these different, reports you see here.

And, ultimately, because we have all of these items in Ketryx, whether they come from ADO, whether they come from GitHub, whether they come from TestRail or Jira, and those can all be connected in a single project if you want, we're able to generate the version specific documentation. I know this requirement is in this version. I know all the, information associated with it. And if I click this button, I'll generate all these documents, and I'll wait for the system requirement specification. And now I have a, collection of all my items.

And you can see here all my requirements. Here's the requirement we were looking at we're looking at. There's the change I made. Of course, we have the the current version of the record at all times. It's also kind of version specific as well.

So this is the version of the item that is effective for two point two. And, of course, this, process is, as you could probably guess, our most popular feature, because getting a first draft of the of the release documentation with just a click of a button is is very, very powerful. And, of course, it's pulling straight from the source system. We have no middleman here. Make the changes in ADO.

Look at the traceability matrix to ensure you're kind of following the process, and then come here and generate your docs. Okay. So, overall, of course, we presented the challenges and and kind of how we consider the solutions within within ADO. But these are the key, questions that you're you you should be considering when you're thinking about, you know, whether how to start using ADO for a sixty three zero four compliant process or how to enhance your current usage of ADO within your process, to obviously release faster without introducing any deviation to that process. The first one is how do we cover the complete total product life cycle?

I showed you kind of how we manage the v model, SEWP, vulnerabilities, risk, software maintenance, even post market surveillance, which I didn't touch on. It's they're all very complicated activities on their own, and we need to link them all together to form a total product life cycle. How do we check all risk controls are executed? That's kind of a one step away, traceability, which can be confusing. Like, I can look at a risk and see it has a control.

How do I look even one step away to see that risk control has a test and then another step to see the test is executed in the version? How do I make sure that everything is under control? How do I make sure everything is approved before, the software is verified and validated? How do I look at historic records when the auditor shows up? How do I, you know, produce my release documentation?

And then how do I connect software item soup and resulting cyber risks throughout my process, which is kind of one of the biggest challenges we see. And the kind of big challenge that overlies all or overlies all of this is this is what your pro total product life cycle looks like. It has so many tools. So, like, how do I do all of these things, not only just in one system, but in all of these systems and across all of these systems, as we develop new software, as we change existing software, as we do post market surveillance and risk assessment, kind of throughout the total product life cycle. Okay.

That is the end of the demonstration period, and my chance and I know we're at time. Thank you so much for coming if you have to drop, but it's my chance to highlight that we have a get for IEC six two three zero four webinar coming, which I showed you those Git items, in in the project we were looking at. So this will kind of take a deeper dive on those. And we have Milan and and Daniel leading that. Milan is kind of a lead software engineer that considers, how Ketryx ingests items from Git.

And Danielle is a director of client operations, which helped and she helps several mega caps, kind of use the system, but specifically use the system with Git, connections for ICC IEC six two three zero four. So very excited for that session, Tuesday, May sixth. I think there's a link in the chat to sign up if you if you're interested. But, yeah, happy to stick around for any questions. Gabriel and I, will answer them for you.

If you have to jump those, thank you so much for attending. Wonderful. And then as you sign off, you'll be presented with a survey on how we did today, how we can improve, kind of other topics that the group might be interested in. We're always looking for feedback. So appreciate your time today and for all the work that you do to deliver safe innovation.