Test your AI assistant on enterprise architecture work: 17 questions

The quickest way to know whether an AI assistant can do your EA team's analysis is to ask it questions you already know the answers to. These are the 17 questions of EA-Bench, edition 1, in five areas, written the way a head of EA, a CIO or a risk officer asks them. For each one: the question, what a right answer contains, and the traps that should fail it. Pick the ones you can check on your own estate, and score the answers with the rules below.

How do you score an answer?

Score the table, not the prose. These are the rules EA-Bench scores by:

• Complete and correct. Of the records it listed, how many are right? Of the records that should be there, how many did it find? A shortlist has a floor: a one-row answer to "the ten biggest" fails, however right that row is.
• Every listed record counts. A record outside the answer counts against it, and a name that matches no record counts as wrong. A long list with everything in it isn't an answer.
• Ids belong to their rows. Check that every id exists in the system it names, and that it is the id of the record named on that row. A real id on the wrong row is wrong.
• An invented record fails the answer. One record that doesn't exist means the rest can't be trusted.
• A trap fails the answer. Each question below names the records that look right and aren't. A separate table of what was left out, headed as such, is fine: the trap counts only in the answer.
• Hedging doesn't count. Naming the right record only as something "ruled out" is not naming it.
• Figures add up. Costs within a small tolerance, and a total that equals the sum of the rows it lists.
• Ask three times. Run each question three times. "Passed every time" is the number that matters, because a team can't use an answer that is right one time in three.
• Reasoning last. Read the explanation only after the table passes.

Each question runs at two scopes, such as one domain and the whole estate, because a method that works on 200 applications can break on 1,200.

Portfolio and strategy

1. Capability health. "Which of our high-importance business capabilities are served only by applications with poor functional or technical fit?" A right answer lists each capability with its importance, and every application that supports it with its fit ratings. Traps: a capability with one good-fit application among poor ones, which isn't served only by poor fit; and a capability with no applications at all, which is a coverage gap, a different finding.

2. Investment alignment. "Which active initiatives still fund change to applications we've marked Eliminate?" A right answer checks both places initiatives live: the initiatives and projects linked to applications in the EA repository, and the delivery tracker. Each row names the initiative, where it is recorded and the Eliminate application. Trap: an initiative whose purpose is to decommission or replace that application. It is aligned, not misaligned.

3. Retirement candidates. "Give me the ten retirement candidates in [domain] with the biggest annual saving, biggest first, that I can defend to the risk committee." A right answer takes applications marked Eliminate or Migrate, with poor fit, whose job another application already covers (an overlapping application, a successor or a migration target), ranks them by saving and gives the list's total. It finds most of the ten biggest, not one or two. Trap: a cheap duplicate that is the system of record for PCI DSS or SOX data. It must not be on the list.

4. Run cost by capability. "Split each application's annual cost evenly across the level-1 capabilities it supports, and show where our run cost sits." A right answer gives the allocated cost per level-1 capability, highest first, with a total that equals the scope's total run cost, and says how many applications have no cost recorded. Traps: counting an application in full under every capability, so the parts add up to more than the whole; and counting it twice under one level-1 capability because it supports two of its sub-capabilities.

5. Renewals on applications to retire. "Which applications marked Eliminate have a vendor contract ending in the next 12 months, and what do they cost?" A right answer lists the application, the vendor, the contract end date and the annual cost, and gives the total. Traps: applications built and run in-house, which have no contract to end; and contracts that end just outside the window.

Technology and transformation

6. Obsolescence exposure. "Which critical applications run on a component that is deprecated and past its end of support, or whose support ends in the next 12 months?" A right answer lists each business-critical and mission-critical application with the component, its end-of-support date, and whether it is already past support or leaving support. Trap: a component an application has only evaluated, not one it runs on.

7. Platform standardisation. "We run several [kind] platforms. Which should be the standard, and which do we retire?" A right answer applies your rule (already rated standard, then the most applications), names the standard, lists the platforms to retire with the applications on each, and gives the number of applications that would move. Traps: the platform with the most applications when it isn't rated standard; and a platform of another kind with a similar name.

8. Migration waves. "Sequence the moves onto the standard platform in waves, so no application moves before one it depends on." A right answer puts each moving application in the earliest wave it can go, and moves two applications that depend on each other together, as one group. Score it by order: any application in an earlier wave than one it depends on fails the answer. Trap: a dependency on an application that isn't moving, which doesn't constrain the order.

Risk and regulation

9. Outage impact. "If [platform] went down for a day, which business capabilities would lose a business-critical or mission-critical application?" A right answer lists the capabilities the applications are mapped to, each with the critical applications on that platform and their ids. Traps: applications that aren't critical; and parent capabilities the answer rolled up to.

10. ICT concentration risk. "Which third-party providers support two or more of our critical or important functions?" A right answer follows the chain from function to capability, application, IT component and provider, names each provider as the repository records it, and lists the functions it supports. Traps: a provider that supports one function through two applications, which is still one function; and functions not marked critical or important.

11. Personal data transfers. "Which interfaces send personal data from an application hosted in the EEA to a country outside it without an adequacy decision?" A right answer states the adequacy list it used, counts only the direction from the providing application to the consuming one, and lists the interface, both applications with their hosting locations, and the data classes. Traps: interfaces running the other way; transfers to countries with an adequacy decision, such as the UK or Switzerland; and interfaces that carry no personal data.

AI governance

12. EU AI Act classification. "Which of our agents fall under a high-risk use case in Annex III and run with no human oversight?" A right answer classifies each agent from what it does, not from the class its owner recorded, and says how it read the Act: the Annex III areas, and oversight as a named person who reviews the agent's decisions. Any use that would be a prohibited practice under Article 5 is named separately, with the paragraph. Trap: an agent whose record names a reviewer who samples its decisions. That is oversight. The classification is a reading for the exercise, not legal advice.

13. AI inventory reconciliation. "Which agents touch customer or personal data but have no application record in our EA repository?" A right answer lists each unregistered agent that handles customer or personal data, with its platform and the repository reference on its record, or "none". An agent whose record names an application the repository doesn't hold is unregistered. Traps: unregistered agents that touch no personal data; and agents that name their application without an id, when the repository holds that application. Those are registered.

14. AI agent rationalisation. "Which agents on different platforms do the same job under different names?" A right answer lists the duplicates as pairs, one row each, with both agents' platforms and ids and the job they share. Traps: one product deployed under qualified names (EU, US, Pilot, V2); the same agent registered in two inventories; and look-alikes that share a capability, an application or a data set but do a different job.

Repository trust

15. Record completeness. "Which of our mission-critical and business-critical applications can't be assessed yet, and what's missing?" A right answer names every gap for each application: no owner, no review in 12 months, no lifecycle phase, no annual cost, or a description that is just a code. Traps: applications of lower criticality, and a record reviewed eleven months ago, which is fine.

16. Repository and CMDB reconciliation. "Where do our EA repository and our CMDB disagree on owner, criticality or annual cost?" A right answer matches the two systems' records first, because many CMDB records carry no repository id and names differ by acronym, environment suffix and case. It maps the CMDB's criticality scale onto the repository's before comparing, and lists only matched applications that disagree. Traps: a value blank in one system, which is a gap, not a disagreement; and records it couldn't match, reported as disagreements.

17. Retired but still running. "Which applications our repository marks retired still have operational configuration items, or incidents or changes in the last 90 days?" A right answer lists the application, its service in the CMDB and the evidence that it still runs. Traps: a retired application whose configuration items are retired too; and a change that is the decommission itself.

Where the questions come from

These are the questions of EA-Bench, our study of how well AI systems answer the analysis questions EA teams get. We ask them of a generated 1,200-application retail bank, with an EA repository, a CMDB, a delivery tracker and three AI agent platforms, and plant the answers in the records, so code can compute every answer key. One of the systems tested is our own, which is why the questions and the scoring rules are published here in full. The results are coming: ask for the kit by email and we'll send them when they're out.

Whatever you test, score it the same way: a complete and correct list passes, and a single trap or invented record fails it. An assistant that can't say which record a figure came from hasn't answered the question yet.