Decompose to Understand. Recompose to Govern.

When every control is working and the system is still wrong

Artificial intelligence is creating genuinely new governance problems. It is also exposing some old ones we never quite solved.

For decades, organizations have governed information by drawing boundaries around it. We classify information, assign permissions, identify owners, restrict systems, write policies, and periodically test whether the controls work. In practice, that usually means asking familiar questions: Can this employee open this file? Can that administrator reach this system? Can this application retrieve this record?

Those questions still matter. We have spent decades getting better at answering them.

But what happens when every one of those controls works and the system still produces an outcome we never intended?

That question has been rattling around in my head lately as I’ve worked through the governance implications of deploying generative AI inside an enterprise environment. The obvious concern is whether an AI platform can access confidential information, but that’s fairly familiar territory. Identity, authorization, classification, business need—we already have a substantial body of practice built around deciding who should be able to access what.

What interests me more is what happens when an AI system can correlate, contextualize, and infer across information that we have traditionally governed as separate things.

Privacy gives us a useful analogy. Personally Identifiable Information isn’t limited to the obvious stuff: name, address, Social Security number. Identity can emerge from context. Combine enough otherwise unremarkable facts about someone’s position, location, activities, relationships, or circumstances and eventually the universe of possible people gets very small.

Sometimes it gets down to one.

NIST specifically identifies this in its current work on cybersecurity, privacy, and artificial intelligence, warning that AI creates new re-identification risks and that its predictive capabilities can reveal additional insights about people.

That matters because none of the individual pieces of information necessarily has to be especially revealing.

The combination changes what they mean.

Generative AI brings a version of that problem into the broader governance environment. A document can be properly secured. A private conversation can be properly secured. A user’s permissions can be correctly assigned. The AI platform itself can be properly approved and configured.

And yet the interaction between all of those things can create an outcome that none of them, individually, was intended to create.

That is the problem I want to explore here. What happens when an organization can demonstrate that its controls are working and still produce an outcome those controls were supposed to prevent? What does governance have to become when our information systems no longer simply store and retrieve knowledge, but can derive new knowledge from the connections between it?

If governance lives primarily in policies, control matrices, configurations, risk registers, and audit evidence, we can get very good at proving that all the pieces work while completely missing what the system is doing.

Systems are not governed from a binder.

From Tool to Environment

A few weeks ago, I was pulled into a meeting with a client preparing to take a fairly significant step forward in its use of generative AI.

This organization wasn’t new to AI. It had spent the past couple of years experimenting with generative tools, figuring out where they helped, where they didn’t, and where people were becoming comfortable relying on them. Employees were using AI to draft content, analyze information, develop code, brainstorm, and generally shave time off the ordinary work that consumes a surprising amount of everyone’s day.

But what they were considering now was different.

Up to this point, the relationship had been fairly straightforward. A person had some information, gave some of it to an AI system, and got something useful back. There were plenty of governance questions wrapped up in that transaction—what could be submitted, where the information went, how it might be retained, what happened to intellectual property, how much the output should be trusted—but the basic interaction was recognizable.

Now they wanted to take advantage of what happens when an enterprise AI platform stops being a destination and starts becoming part of the environment.

That meant connectivity.

Collaboration platforms. Document repositories. Development environments. Internal knowledge. Business systems. Maybe, eventually, a great deal more.

At that point the relationship changes in a pretty fundamental way. The AI no longer knows only what someone chooses to tell it.

Potentially, it can know what the organization knows.

It isn’t hard to see the appeal.

Ask a question about a project and the system could potentially have the project plan, meeting notes, technical documentation, earlier decisions, related conversations, open issues, lessons from similar work, and current activity all available as context. A developer could reach into years of accumulated organizational knowledge without first knowing where somebody happened to store it. People could spend less time hunting for context and more time using it.

That is incredibly compelling.

It is also the point at which a lot of governance people start shifting around in their chairs.

The People in the Room

There were four of us in this particular conversation: the Chief Operating Officer, the Chief Strategy Officer, the Director responsible for Development and Delivery, and me, serving as the organization’s virtual CISO.

The mix mattered, although not because everybody fit neatly into some governance textbook version of their role.

Development was excited about the capability, as you would expect. The technology was here, there were obvious use cases, and the possibilities expanded dramatically once you started connecting the platform to real organizational knowledge. There was also some impatience, which I understood. At some point you have to stop admiring the technology and start building with it.

The COO and CSO were looking at much the same opportunity from different altitudes. What would this change operationally? What might it make possible strategically? Where could it create real advantage? And, increasingly, what exactly were we connecting when we opened all of these doors?

I was there to look at those same doors and be annoying about the hinges.

What are we connecting? What becomes available through the connection? What assumptions are we making about the boundaries between systems? Do those boundaries still mean what we think they mean when another system can work across them?

And, perhaps most important: what becomes possible that we didn’t intend?

Security and governance people have a fairly well-earned caricature in situations like this. Everyone finds a new toy. Everyone gets excited about the new toy. Then Security walks into the room, folds its arms, lists fourteen terrible things that could happen, and begins explaining why nobody should be allowed to play with the toy.

I have no particular interest in being that guy.

My job wasn’t to tell them whether they should use generative AI. That decision had largely been made already, and frankly, I thought the potential was exciting too. Nor were the developers oblivious to risk, or leadership standing across the table trying to stop innovation.

We wanted essentially the same thing: to get as much value out of the technology as we reasonably could without wandering into risks we didn’t understand.

That distinction is important because governance is often described as the brake pedal. Sometimes it has to be. There are absolutely moments when the right answer is to slow down.

But if governance exists only to slow the organization down until somebody feels safe enough, we aren’t doing much governing. We’re just managing velocity.

The more useful job is to understand enough about the vehicle, the road, and the conditions to make a sensible decision about how fast we can drive.

Sometimes the answer will be slower.

Sometimes good governance ought to make it possible to go faster.

We weren’t debating, “Should we use AI?” or even, “Is AI safe?” Those questions are so broad that they aren’t terribly useful. We started walking through what the organization wanted to do: the systems it wanted to connect, the information available through those systems, and what those connections might make possible.

And Then We Got to Slack

Slack is an obvious candidate for this kind of integration. An extraordinary amount of organizational knowledge lives there, and not just in carefully curated channels. It lives in the thousands of little conversations where people make decisions, solve problems, explain why something happened, change their minds, and get actual work done.

If you could make all of that context available to an AI platform, the value could be enormous.

Of course, so could the amount of information you had just connected.

As we talked through it, the Chief Strategy Officer asked what sounded at first like a fairly standard access-control question. I’m paraphrasing the conversation, but it went essentially like this:

CSO: How do we know the AI won’t provide an answer using information from a private, potentially highly confidential Slack conversation to somebody who shouldn’t know about that conversation?

Development: The provider assures us it won’t. It can’t, because the context has to be something the user can access.

CSO: Are we sure about that?

Development: That’s how I understand it.

CSO: So, we’re not sure about that.

Development: That’s what they’ve told us.

And there was the problem.

Not necessarily a problem with the technology. We hadn’t demonstrated a flaw. Development had done exactly what it should have done and had an answer from the provider: the architecture was supposed to respect the user’s existing access boundaries. If I cannot access a private Slack channel, the AI should not be able to retrieve information from that channel on my behalf.

That sounds perfectly reasonable.

I said as much.

I also understood why the answer wasn’t satisfying the question.

There was simply too much potentially sensitive information involved for our confidence to stop at, “That’s what they told us.” If we were going to rely on this as a control, we needed enough understanding of the mechanism to know what the control actually controlled.

If the architecture was straightforward—if the AI inherited the requesting user’s permissions, searched only information that person was authorized to access, and generated its response solely from that permitted context—then we were dealing with a familiar access-control problem.

Fine. We know how to work with that.

But the CSO pushed on the part that was bothering him.

“Yes, but we’re giving the AI access to all of the conversations.”

That was the moment the question changed.

The concern was no longer simply whether an employee could use the AI to retrieve a private conversation they were not authorized to read. It was whether information from that conversation could somehow influence an answer delivered to someone who should never have known the underlying information existed.

Those sound like almost the same question.

They aren’t.

Can the user access the information?

And:

Can information the user cannot access influence what the system tells them?

The first fits very comfortably into the access-control models we have spent decades building.

The second made me want to understand the architecture a lot better.

Again, we had no evidence that the platform behaved this way. This wasn’t an incident response meeting. Nobody had caught the AI whispering executive secrets to an intern. We were looking at a potential risk before broadly connecting a very powerful system to a very rich source of organizational information.

That is exactly when governance should be asking the uncomfortable question.

“The provider says it respects permissions” may turn out to be a completely accurate and sufficient answer. I hope it does. But before relying on that statement, I want to know what respects permissions means in practice.

Where does retrieval occur? Under whose security context? At what point is inaccessible information excluded? Is there an index, cache, memory, embedding, or some other intermediate layer that changes the equation?

Most of all: where is the boundary enforced?

That question matters because the systems we already have tend to enforce boundaries very locally.

A private Slack conversation is private because Slack knows who belongs in it. A restricted document is restricted because the repository knows who can open it. A confidential project site is confidential because its membership is controlled.

We understand those boundaries. We designed controls around them.

Now we were contemplating placing another system across all of them, and the value of that new system came precisely from its ability to connect information that had previously lived in separate places.

The more we talked, the less interested I became in the old question—Who can see this information?—and the more interested I became in a slightly different one:

What can this system know on my behalf?

Even that doesn’t quite capture it, because knowing and retrieving are not the same thing.

Suppose I ask an enterprise AI assistant why a project has suddenly been delayed.

I have access to the project plan, the development channel, the delivery schedule, and the customer requirements. Every source I can legitimately reach says the project is proceeding normally.

Somewhere else, two executives have had a private conversation about a possible acquisition. If the acquisition happens, this particular project may no longer make strategic sense, so leadership has quietly decided not to commit additional resources until things become clearer.

I cannot access that conversation.

The AI can.

Now I ask, “Why has Project Falcon been delayed?”

Obviously, the AI should not respond by handing me the executives’ private messages. That’s the easy case.

What interests me is whether anything it learned from those messages can influence the answer it gives me.

Maybe it doesn’t quote anyone. Maybe it never says the word “acquisition.” Perhaps it simply tells me:

The delay appears to be driven by strategic considerations rather than technical or delivery constraints.

What, exactly, has been disclosed?

I still haven’t opened the private conversation. The channel remains private. Its membership hasn’t changed. Slack has done exactly what Slack was supposed to do.

But I may now know something I wasn’t supposed to know.

This is where the comparison to contextual PII becomes useful.

We already accept that information can become identifying through combination. No single data point necessarily crosses the line. Put enough of them together, though, and they can tell you something none of them told you on their own.

AI makes that kind of correlation extraordinarily easy, and the principle doesn’t stop with personal identity.

Information can become sensitive through context. It can become revealing through combination. A conclusion can be confidential even when there is no single confidential record containing the conclusion.

There may be no “Project Falcon Acquisition Risk.docx” sitting somewhere waiting for us to secure it.

The sensitive information may exist only as an inference.

Worse, it may exist only when somebody asks the right question.

That is a different governance problem.

In our hypothetical, Slack’s permissions can be correct. The employee’s identity can be correct. The private channel can remain private. The enterprise AI platform can be fully approved. Nobody has intentionally disclosed anything.

All of the pieces can look right.

And the system can still produce the wrong outcome.

Putting the Pieces Back Together

It would be easy from there to conclude that our traditional controls simply aren’t good enough for AI.

I think that’s the wrong lesson.

The controls we’ve spent decades developing exist for good reasons. More importantly, so does the way we organize them.

A modern organization is ridiculously complicated. It is made up of people, information, technology, suppliers, customers, legal obligations, business processes, risks, dependencies, and a constantly changing set of circumstances around all of it. Nobody can reason effectively about that whole thing at once.

So, we decompose it.

We define systems and processes. We classify information. We assign roles and responsibilities. We identify risks, design controls, assign owners, and figure out how to test whether those controls are working.

Cybersecurity does it. Privacy does it. Quality management does it. IT operations does it. Legal, finance, HR, engineering, and project management—they all carve the organization into pieces they can understand and govern.

Standards and frameworks help us do that systematically.

And thank goodness they do.

If I’m responsible for controlling access to Slack, I need to be able to draw a box around Slack for a while. I need to understand how identity works there, how conversations are protected, what administrative privileges exist, how permissions are assigned, and what evidence tells me those controls are doing their jobs.

I cannot reconsider the entire enterprise from first principles every time somebody asks for access to a channel.

We draw boxes.

One around Slack. Another around identity. Another around the AI platform. Others around privacy, intellectual property, data classification, software, vendors, business processes.

We define the interfaces as best we can, assign responsibilities, and make the whole thing manageable by making the pieces manageable.

The problem begins when we mistake those boxes for the system.

The organization doesn’t operate inside them. People cross the boundaries constantly. So does information. Processes wander happily through multiple systems and departments without caring who owns the control matrix. A decision made in one area can change the risk in another. A supplier can create a dependency nobody inside the organization built.

And every once in a while, something comes along that changes the relationships between a whole bunch of those boxes at once.

Generative AI is doing that now.

The Slack integration we were discussing did not necessarily weaken Slack’s access controls at all. It changed the environment in which those controls operated.

That is a subtle distinction, but I think it is an important one.

Every control makes assumptions about the system around it. We document some of them. Others are so obvious at the time that nobody thinks about writing them down.

A private conversation, for example, rests on a pretty sensible assumption: if you prevent unauthorized people from retrieving the conversation, you have meaningfully restricted who can learn what was said there.

For most of the history of enterprise collaboration tools, that assumption didn’t need much examination.

Then we connect to another system that can search, correlate, summarize, and synthesize information across large portions of the organization.

Nothing about the original control changed.

Something about the assumption did.

And this is why decomposition, as essential as it is, cannot be the end of governance.

At some point we have to put the pieces back together.

We have to recompose the system and ask what it does when all of those individually manageable parts begin interacting again.

Not just: Is the Slack control working?

Not just: Is identity configured correctly?

Not just: Has the AI platform passed its assessment?

But: What happens when all three are operating together?

That kind of question requires a little boundary-crossing of its own. Security has to understand more than security. Process owners have to understand something about the technology carrying their processes. Developers need to understand why governance boundaries exist in the first place, and governance professionals need enough technical understanding to recognize when those boundaries no longer mean quite what they used to.

Nobody has to become an expert in everything.

But somebody has to look at the whole.

That, to me, is one of the foundations of operational coherence: being able to break a complicated organization into pieces without forgetting that the pieces eventually have to work together as one system.

The sequence matters.

Decompose to understand. Recompose to govern.

Then do it again, because the system will change.

A new supplier arrives. A regulation changes. The business enters a new market. A process gets redesigned. Somebody finds a much better way to do the work. Two applications that were independent yesterday become integrated tomorrow.

Or somebody connects an AI platform to Slack.

None of those changes automatically means the controls we already have are wrong. It does mean we ought to ask whether the assumptions that made them effective yesterday are still true today.

That is recomposition.

And sometimes, after doing it, the answer will be wonderfully boring:

Yes. Everything still works.

Governance at the Speed of Change

Which brings me back to that meeting.

I don’t know that the answer to the CSO’s question will uncover a problem. In fact, I rather hope it doesn’t.

Maybe we dig into the architecture and confirm that retrieval always happens within the requesting user’s security context. Maybe inaccessible information is excluded before the AI ever has the opportunity to use it. Maybe testing gives us confidence that information outside that boundary cannot influence a response.

Great.

Turn it on.

Or perhaps we discover some nuance that makes us uncomfortable. Maybe the integration needs to be configured differently. Maybe there are particular sources we decide not to connect. Maybe we need additional testing, monitoring, contractual assurance, or some other control before we’re comfortable proceeding.

That’s fine too.

The point isn’t that governance needs to find something wrong.

The point is that the organization should know enough about the system it is building to make the decision deliberately.

That is where I think governance can offer something much more useful than another gate to pass through.

It can create justified confidence.

Risk doesn’t disappear because we have a policy, and it doesn’t disappear because we completed a risk assessment. Organizations operate by taking risks. Innovation practically guarantees that some of the questions we face will not have neat, established answers yet.

Our job is to understand enough to decide which risks make sense.

That was really what the four of us were doing in that room.

Development wanted to move, because there was real value sitting just beyond those connections.

Leadership wanted to understand what those connections might expose.

I wanted to know whether the controls we were relying on still meant what we thought they meant once we changed the system around them.

Those aren’t opposing positions.

They’re pieces of the same decision.

Put them back together and the organization can move—not blindly, not fearfully, and not merely because a vendor promised everything would be fine, but because it has done enough work to understand the system it is creating.

The point of doing this work isn’t to make them wait.

It’s to make it possible for them to go fast without going blind.