Proof of Concept: How Do We Ensure Ethical AI Systems?

Transcript

Anna Delaney: Hello, this is Proof of Concept, a talk show where we invite security leaders to discuss the cybersecurity and privacy challenges of today and tomorrow and how we can potentially solve them. We are your hosts, I'm Anna Delaney, director of productions at ISMG.

Tom Field: I'm Tom Field, senior vice president of editorial with ISMG. Anna, always a privilege to be talking about AI with our guests today.

Delaney: Tom, this shifting tech progress, the deployment and utilization of AI systems raise important questions about potential negative impacts, vulnerabilities and uncertainties. The questions we're asking today center around identifying and mitigating risks to ensure the ethical and responsible deployment of AI. What are the key aspects of evaluating risks in AI? Are biases and fairness adequately addressed? How transparent are the decision-making processes of AI models? What measures are in place to ensure security and privacy? Tom, before we bring on our experts, anything you want to address on this topic on evaluating risks around AI?

Field: There's a governance topic that's emerging, it's something I'm becoming aware of now more so than ever, there is a whole new insider risk element when it comes to the use of AI. As I talk with security leaders, they're having a hard time getting their arms around not just what their enterprise policies are, for AI use and use of data, but individuals. Just as we found that there are malicious individuals who intend to commit fraud and do harm within an organization, there are insiders who have accidents and get taken advantage of by external parties. AI is a whole new element through which you've got greater insider risk that needs to be governed. I'm hearing a lot more people talking about this. Don't know whether we'll get to it today in this conversation, but it's certainly something on my mind now.

Delaney: That's something I'd like to ask our experts in the room. Speaking of which, let's invite them back to the studio extending a warm welcome to Heather West, senior director of security and privacy services at Venable LLP, and Sam Curry CISO at Zscaler. I thought we could start with the fact that it's crucial to understand the potential weak points in AI systems to ensure robust defenses against threats. My first question is about the methodologies and considerations in assessing this risk. How do you assess the vulnerability of AI models to adversarial attacks? Sam, you want to go first?

Sam Curry: A lot of it has to do with where it's applied and how it's used. There are three places to look at. One is on input, how is the information collected? The second is the model itself. The third is on the output, how's it used? You mentioned in the preamble, the fairness and ethics side of it. We want to be careful that we understand the threat model, we understand the implications societally and on the applications, and who's trying to mess this up. It can be very subtle, and certainly in the case of say adversarial AI, people who are doing AI poisoning, people who are trying to duplicate the models. We can have bias, leakage, malicious intent and inadvertent intent. We should think about things like threats to data and autonomy, we should think about things like bias and discrimination, and we should think about things like liability, but we will get the machines we deserve. Unless we take steps to avoid that or to drive them in the direction that we want. My implication is we need to take steps now.

Heather West: We need to put the effort in it, we need to put the time in and that is the least exciting part of building these systems, but it's important. We don't want to end up in that science fiction novel where they didn't put that effort in. When we're thinking about the risk, there's a broader set of risks that we're thinking about to get to both the cybersecurity and the ethics and responsibility side of things. We're talking about vulnerabilities, but we're also talking about flaws in the system, glitches, and errors. Within all of those contexts you were mentioning, the scope of potential issues that we're looking at and the potential attack surface feels a lot bigger. As an industry, we're grappling with whether that's a true statement, but we're also bringing in all of the ways of assessing vulnerabilities, flaws and system integrity that we do have context for. It's this race to make sure that we have a clear methodology. None of this is new, but there are new pieces of it and as we're thinking about potential attacks, there's a bunch of new pieces of the list as we go through and evaluate.

Curry: I couldn't agree more, and I think we're going to start finding that there's a general leakage happening that has a feel of the uncanny valley. The uncanny valley usually refers to how people look or when things start to look human, almost perfectly, but not quite. This is going to be a more subtle one, we start to see the thinking machine behind it. Is there a thought process happening behind the scenes? That leakage will start to come through. What we want to do is, we want to be careful to be considering these things ahead of time, not after the fact. Some of these leakages are non-reversible, once the data is out there, it's out there. Some of the applications are difficult to reverse. The business relationships around how these things get consumed are being established now. Generally speaking, when you apply AI and ML, the places you can use it and trust it the most are the ones that are deductive rather than inductive, it's when you go from the general to the specific. When it's a cog in the machine, not when it is the machine. Rather than a thinking machine, what we want is AI to be applied to fulfill functions within a wider system, things that are testable and verifiable, and we understand. It does a piece of the job rather than the whole job. We can hold the owners or the producers of it accountable, and the owners and the producers of the data that goes into it and the output accountable. That's something we can test and put limits on rather than "Don't worry, the machine will fulfill the job of the human." Let's use it to fulfill parts of the job that help humans.

Delaney: What practical measures or strategies would you recommend to bolster these systems and bolster the resilience of AI systems?

West: You hire a bunch of people like me. You need someone whose job it is to sit down and say, "How are we designing this?" "What is the data that's going into it both in terms of training the model and the data that's going to get used to in the system day to day?" "How is it being developed?" "How are we training it?" "How are we putting it into practice?" "How are we testing it over time once we do that?" That's something that I think each of those steps is going to have different answers in terms of risk and mitigations, and the steps you can take. However, each of those may have different actors involved. You need someone who sits down and says, "I know enough about how this system works, and I know enough about what it's supposed to do, and I know enough about the context in which it's being implemented and it's being deployed!" To be able to say, "Here are the risks, and here's how we mitigate them." The goal is to get to the point where you can do these impact assessments and say "I understand it well enough, I am comfortable with it." All of our systems are flawed, whether they're AI or otherwise. We're getting to a point where we can tolerate the risk. What we need to do is make sure that we understand the risk well enough to decide whether we tolerate it. We need to get past that black box. AI is cool, AI is fun. But, when we get to leakages, where it's not working the way we think it is or the way we think it should be, we need to be diving in and saying "Okay, let's figure it out." That process is going to differ a lot depending on what kind of AI it is. If it is, as Sam was talking about, a deductive system that says, "I'm looking for this general thing, I found it, here's a small example." That's relatively easy to test. If you instead are talking about testing LLMs and whether they're behaving correctly, that's hard and you're doing a series of spot checks. As a measurement science and as an evaluation science, we're going to get better at that over the next year or two, let alone the next decade or so. However, right now a lot of this is making sure that you've done your due diligence.

Curry: I agree with you, Heather. I would go a step further with some general principles. First, we can be open and transparent about how we do the governance of this, we can also bring in more people from outside. Hire the Heathers and then bring in people to give you advice as part of a committee or a board or governance. We do this in medicine. Before someone does oncology research, there's a group that says, "Are you doing it right?" "How are you doing it?" This can slow things down, but if it's done correctly, it should speed up research because of this notion of peer review and guidance. Brilliant scientists don’t say, "I'm going to do this research," they have the support of other scientists who say this is the direction it goes, and people can build on that work. That doesn't mean that you're exposing the IP, or what's generated from it, but it is that the work is something that can be seen. We can get specific about what the risk models are. I talked about collecting data, building a model and productizing the model. Now we can say, "What are the malicious or adversarial things we're worried about and list them?" Then say, "What's the risk of each of these and put them in a document?" We would then say "What are the countermeasures are the controls?" as we would in security in a classic sense, that makes it one of the privacy concerns; another list. What are the fairness and ethics concerns? Another list. There are technological answers, and those have to have programs around them. How effective are they being? How can we improve them? By measuring and quantifying it, we can get better over time. It's not a "we're concerned how generative AI will revolt!" Those are philosophically very important, but we can get specific and operationalize this stuff. Hire Heather, go get an advisory board, and then start to manage it just like you would any other form of risk.

West: Then you can start to map this out, and the documentation is important. However, saying "Here are the vulnerabilities that we're worried about, how are we mitigating them? What controls we're putting in place? Here are the risks we're thinking about, because those are different than vulnerabilities a lot of the time, especially when we're talking about AI, and here's how we're putting controls in place. Here are the threats we're worried about; here's how we're putting controls in place. Breaking it down into a taxonomy makes it much more manageable, and you have a list of things to do. I don't want to suggest that it is checklist security, but you have a checklist to go through. It won't be perfect, but it will make a lot of progress.

Field: Sam, you said get specific, so let's. Let's talk about data privacy and bias, and at the moment I direct my question to you, Heather jump in as well. What are some of the methodologies you see now to be effective when it comes to evaluating and mitigating the privacy concerns in gen AI systems?

Curry: In this case, there are a few different approaches. Generative AI and LLM are fundamentally language modeling and manipulation and re-slicing. What you apply it to matters and makes a lot of difference. We're going to see improvements there, but the question now is very often about copyright, idea provenance, and intellectual contamination. We've had these issues for a long time, for the most part. However, we didn't ask these questions when it was human beings. You go to college, you study for four years, you come out, and then you write something? Did your brain take ideas and push them together and are those ideas yours? It's different when the machine does this in a matter of seconds. We have to have the jurisprudence. We have to have a way of interpreting this. I teach a number of courses and I tell my students, "Cite your sources, and then go check what the LLMs produce." The other thing is how this gets used in a business context. We have to have the contracts come about, we need to know where the liability lies and we need to understand the implications. Did you accidentally doxx someone? Are the sources real? Do they mean what you think they mean? You have to take accountability as a "prompt engineer." When you're doing this work, you are responsible for the output, you can't just abdicate that. You've got to say "Now at the crafting of the writing, and the putting together of it into a coherent whole," that's easy now. You are now responsible, and most of your work has to go into validating that it is what you mean to say and that it is accurate and true. You may get into bias. You can have something called the collector bias. Are the sources truly well distributed? And is it good research? Don't abdicate that, right? You can't just give that up to the LLM unless you understand how the LLM was doing it. You get leaky abstraction and modeler bias. Look at the application. Is this being used to write a white paper? Or is this being used to decide what sentencing is in court cases? Those are different things? In some cases, we may want to say, you can use it for one, you can't use it for the other. Then there's the technology controls that were put in place around it. It's important to understand that we can't rely on detection technology to find it, we're going to have to put AI in place to make sure that what you're producing is statistically likely and isn't either suffering from bias or adversarial injection. If somebody has an agenda for whatever application you're using it for, they aren't injecting a bulk of data that is going to make the AI go in a direction you don't expect. You'll believe the output because that's what the machine is telling you. Things like federated learning, where you look at multiple processing at edges of the data coming in and say "Okay, is it coming in the same statistically from everywhere and should it? Or it is being done in parallel with a different system, instrumenting statistically valid samples in parallel? You then compare the results. These are ways you can watch out and make sure is it processing the way I think it is, or is it being subjected to influence. Heather?

West: There's a huge number of places that we can inject a little bit of thoughtfulness into this problem, particularly around privacy and bias, which is already like a very abstract difficult problem. There are a lot of definitions of privacy, bias is a little bit more straightforward, if difficult. I want to re-emphasize a piece here, Sam was talking about what the system is used for. The easiest simplest thing you can do to protect against leakage, data leakage, privacy and bias is to make sure that you're using the system for what it's supposed to be used for. If you asked one of these general-purpose LLMs, all sorts of questions, you can come up with bias easily because it's trained on biased data. If what I'm doing is writing a joke about my dog, that is fine, it's not impacting people. It's not great, but let's figure out how to fix it. However, if it's saying that all of the men get into med school and the women don't, then maybe let's not apply the LLM to med school admissions. Making sure that it is fit for purpose does a lot in terms of bias until we have it figured out how to build systems that do what we need. That said, I don't want to imply that we don't know how; I want to imply that it is harder. There are statistical disciplines that are massively more popular than they were a year or two ago, around trying to figure these things out. Sitting down and talking to some of the folks who are in the industry, who have seen regulations in terms of bias for decades it is one of the things that is useful. How do you make sure that you don't have unintended bias when you're determining who gets a mortgage, or who gets a job? We can start translating that into other systems so that the joke about my dog isn't biased in weird ways. However, that's going to take a little bit of time, and it's going to take a lot of thoughtfulness.

Curry: We know how to do it, but it's more important that we know how to approach it. What we want to do is - over time - we want to establish the means by which we improve. We don't know how to do the problems that will exist in three years that are derivative of the work we're doing now. It's less about getting it right now forever than setting up the boundaries by which we're going to continuously get better at this and how we're going to avoid the disaster scenarios. For example, the application of LLM in my role towards finding malware, spotting fraud, or looking at data classification is very different from the use cases Heather was saying around mortgage selection; interest rate determination; or something in a legal or medical context. Who gets what kinds of services or what kinds of what kinds of sentences? Those things are not the same, even though they use the same technology. There's going to be a desire to say, "Let's get the rules right once, and let's apply it everywhere." It is not going to turn out that way now, and it's going to get even more diverse. For lack of a better term, there will be an unnatural selective process over time, where our human society is going to drive speciation in different behaviors, in the governance of these things, based on the application. We're going to get an ecosystem of AIs, and how they're ruled over and behave and how they interact with our human society is going to turn out very different based on the application and how far down that tree you go.

West: It's all about governance.

Curry: It is.

Field: Heather, how can bias in training data and model predictions be identified and addressed? What are you seeing work?

West: Start with the training data, you can do a lot of work on the training data. Before you look at the data, ask where's the data from. In a lot of these datasets, we're discovering that there was a real self-selection issue. If you're talking about an engine running a facial recognition model, and you bring in a bunch of this data, and then you realize that the way you got it was a bunch of 20-year-olds at a college got paid 20 bucks to sit in front of a camera, that is a very different population than you are necessarily thinking about for the general usage of these models. All of those students are on the West Coast in the United States, and you're even more specific. You say, "Where's my data? Where did it come from?" You then start looking at the data and saying, "Does it say what I want it to say?" Because they're good training data sets, or that they seem good, that are embodying some bias that you didn't intend to put in there. There are some interesting examples of sentencing data with high-impact use. They said, "We're going to make the assumption that our court system and our judges are fair." I don't think they said that explicitly, but that was the assumption they made. Then they went back and they said, "When we do the analysis, this isn't fair, and we're magnifying the bias issues that were already present." We've seen the same in resume review. As an alum of women's college, we often get hit with that one in the tech industry. The model would look at my resume and say, "Oh, she went to a women's college, we don't hire that." Part of it is looking at the training data, but then the flip side is part of it is looking at the outcomes and saying, "Is this what I expected? Maybe my resume isn't amazing, but maybe another resume, should have flown through the process, but just didn't get flagged for further review?" Looking at it and making sure there's an unfortunate amount of gut checks in the process right now, but that's important. A lot of these things are found because someone looks and says that doesn't seem quite right. As we build that into a more robust process, and a more robust governance, we're starting to talk more about AI impact assessments. We're starting to talk about bias evaluations, we're starting to talk about bringing that disparate impact analysis to the table. Right now, we're still establishing the best way to do that. I think there are a lot of processes underway in the U.S. and globally, trying to decide how do we make this measurable. How can we demonstrate progress with numbers, which is going to be important? A important progress is to be able to say "I made it better." However, a part of that is just looking at every step in the process, design, training, data, use and making sure that it does what you think it should be doing.

Field: Sam, your thoughts on bias?

Curry: First, maybe process-wise, you need other people examining the thinking. It's not an audit, it's to go in and suggest improvements to the collection process or say, "We'll look for those biases, not for the purpose of saying, 'Hey, you're an idiot, you didn't do it right.' It's to go through and say, oh, there's an unconscious bias in the data, or you probably didn't think about it, but the people you could reach and get to take part in the program, they're not demographically diverse for the application it's going to be put to." Alternatively, you collected data from one courtroom which has these demographic differences from where it's going to be used. Your mention of colleges made me think we may need the liberal arts more than ever before. We've talked about STEM forever, and my brother uses the term STEAM now. He says we got to put arts in STEM. We need philosophy, we need ethics. We need things like archival sciences, and history because the data itself is important and how it's preserved and how it's managed. We need to be looking at things like literature. What is the art output? The post-moderns used to say, if a camera takes a perfect picture why do we need a painter anymore? Now we've got all the arts affected by this, but also sociology and psychology. How does it impact humans in the wild? those are sciences that are typically seen as more social sciences. This isn't just the "hard sciences" anymore. There are rules for that, and this applies to the input and output of the models as well. We need all those checks and balances here in the system. Let's not denigrate the "What can you do with an English degree?" question anymore.

West: That also brings up a piece of the puzzle. To the extent possible, these systems need to be designed by people that reflect the user base and reflect the context in which it is used. That is a bit of an ambiguous statement. However, we have seen this before, when a bunch of incredibly well-intentioned engineers - who come from the same background - build a system, they tend to overlook important things that matter to the rest of the world. Sometimes it's comical how that comes out. Over the last 15 years, I have watched the tech industry come to that realization that they need to think beyond themselves, we need to continue that it is worth explicitly saying that the AI systems that have some of these unintended biases and impact people in unexpected ways specifically need to be designed by diverse communities.

Curry: Anybody from the tech industry listening to this and shaking their head and going "Now you're just exaggerating," this is a path to more money for you. If you want society to be embracing these things, and open up new markets, this is the way to do it. Let's keep in mind that the AI technologies that we're building don't think we do, and we tend to think they do. There's an article in The Atlantic about Go playing. We learn through narratives, we interpret through narratives, and we tell stories. AI doesn't do that, and that's different. When it goes through, and it finds the pointers to what's relevant, that satisfy us, we go and put a narrative on top of it. That's our role in the system, we're not just the consumers of it, we're the ones that are going to put an interpretation on it. It is not inherently there from the machines that elicit the patterns and the trends and the things to notice. It's our job in the system to make sure that the right narratives are the ones that are being continued or being met or not met. In this Go story, AIs were giving Go masters new games, and the Go masters would watch these initial plays and say "What are you doing? It doesn't make sense, these moves logically don't make sense." They weren't getting beaten, they were getting trounced. Normally, they lose by a few stones, but they don't lose by a lot. However, they were, and in retrospect, people were saying it was like watching aliens play the game. It was like a gift from the future. They're analyzing this stuff because they can't put narratives to it. Another example is the AI that was famously looking to find ways of recognizing fish in pictures. The way it found it was by fingers because almost every picture it was trained on had people holding the fish. Humans wouldn't do that; we'd be looking at the fish. We're going to get better at this over time, but only if we do this in a conscious and deliberate way.

Field: Excellent conversation, as always, we've got more topics than time, it seems.

Delaney: We've had a wonderful time listening to you both. We want to talk about explainability and transparency and how we understand these decision-making processes and regulations. Maybe we'll save them for another time. Thank you so much, Sam and Heather; this has been informative and educational. We appreciate your time.