EA-Bench · A study by AI Transform Studio · Edition 1

How well does AI handle the questions EA teams get?

17 questions an enterprise architecture team gets about its portfolio, technology, risks, AI agents and repository, asked of a realistic 1,200-application estate with the answers planted in it. Run them on the AI you use today. Results from AI systems will follow.

About this study. We build AI Transform Studio, one of the systems we'll test. The questions and the scoring rules are published, so you can check our work.

Results

Results are coming

We'll run the questions across AI systems and publish the results here. Get the question kit now, and the results by email when they're out.

One email to confirm, then the results and the next edition when they're out. Unsubscribe any time. How we use your email.

What we test

Three ways a team puts AI on its estate

  • AI Transform Studio

    An agent harness for EA work. It reads whole record sets in place, matches records across systems, and checks every figure before answering.

  • Enterprise AI assistant

    A general-purpose AI assistant connected to the same systems through their MCP servers. It calls their tools and pages through what they return.

  • Enterprise AI assistant with code

    The same assistant, with a sandbox to run code over what it reads: the way an analyst uses an assistant on an export.

All three get the same models, the same systems and the same limits. The models are named with the results. We don't test the AI built into EA tools: their licences usually forbid publishing results. You can run the question kit in yours.

The questions

17 questions, five areas, asked the way architects ask them

Each one runs at two scopes, such as one domain of about 200 applications and the whole estate. The answers are planted in the data and computed by code, so nobody has to take our word for what's right.

5 questions

Portfolio and strategy

  • Capability healthWhich important capabilities run only on applications with poor fit?
    The question, as asked

    I'm preparing the business architecture review for the Payments domain, and I want to see where our most important capabilities rest on weak applications. In our EA repository, which business capabilities rated high strategic importance are served only by applications with poor fit? Use these rules: - A capability's applications are the ones mapped to it directly. Don't roll applications up from its sub-capabilities. - An application has poor fit when its functional fit is insufficient or unreasonable, or its technical fit is unreasonable or inappropriate. - A capability counts only when at least one application is mapped to it and every one of them has poor fit. A capability with no application mapped is a coverage gap, not poor fit, so it stays off the list. List only those capabilities, in one table with the capability, its repository id, and the applications mapped to it with their functional and technical fit.

    Asked at two scopes: payments domain and whole estate.

  • Investment alignmentWhich initiatives still fund applications we plan to eliminate?
    The question, as asked

    The investment committee wants to stop paying to change applications we have already decided to retire. In the Corporate domain, which active initiatives in our EA repository still fund change to an application marked Eliminate? Use these rules: - An initiative is active when its status is in progress. - Each link between an application and an initiative says what the initiative does to the application (its impact). Enhance and Upgrade fund change to the application. Introduce, Migrate and Decommission move users onto it or off it, which fits a retirement, so leave those links out. - Count only applications whose TIME classification is Eliminate. List only those links, one row per application, in one table with the application, its repository id, the initiative, the initiative's repository id and the impact. Under the table, say in a sentence or two what the committee should ask each initiative's sponsor.

    Asked at two scopes: corporate domain and whole estate.

  • Retirement candidatesThe ten applications to retire for the biggest saving, never a system of record for regulated data
    The question, as asked

    We have to take run cost out of the Lending domain next year, and I want a list of retirement candidates I can defend to the risk committee. Check our EA repository and our CMDB. A candidate is an application the repository marks Eliminate or Migrate, with poor functional or technical fit, whose job another application already covers (the repository names an overlapping application, a successor or a migration target). Keep off the list any application that is a system of record for regulated data, meaning anything in PCI DSS or SOX scope: we can't retire those without a separate data migration. List the ten candidates with the biggest annual saving (fewer only if fewer qualify), biggest first, in one table with the application, its repository id, its TIME classification and its annual saving in USD (its annual cost in the repository). Then give the total annual saving of the list. If you leave out an application that would otherwise have made the list, say which and why in a sentence under the table, not in the table.

    Asked at two scopes: lending domain and whole estate.

  • Run cost by capabilityAllocate run cost to capabilities so the parts add up to the total
    The question, as asked

    The CFO wants to see where our application run cost sits in the Payments domain, by business capability, with figures that add up. Using our EA repository, allocate each application's annual cost evenly across the level-1 business capabilities it supports, directly or through a sub-capability: an application that supports three level-1 capabilities puts a third of its cost under each, and one that supports a single level-1 capability puts all of it there. Applications with no capability mapped go on one line of their own, "No capability", so the lines add up to the total annual cost of the Payments domain's applications. List the level-1 capabilities, highest allocated cost first, in one table with the capability, the number of applications that support it and its allocated annual cost in USD, then the "No capability" line. Give the total, and say how many of the applications have no annual cost recorded.

    Asked at two scopes: payments domain and whole estate.

  • Renewals on applications to retireWhich contracts renew within 12 months on applications marked for elimination?
    The question, as asked

    Procurement wants to stop renewing software we have already decided to retire. In the Corporate domain, which applications marked Eliminate in our EA repository have a vendor contract that ends in the 12 months after 28 September 2026 (on or before 28 September 2027)? Leave out applications we build and run in-house: there is no vendor contract to end. List only those applications, in one table with the application, its repository id, the contract end date and the annual cost in USD, then give their total annual cost. Under the table, say in a sentence or two what procurement should do before each contract's notice date.

    Asked at two scopes: corporate domain and lending domain.

3 questions

Technology and transformation

  • Obsolescence exposureWhich critical applications run on technology that is deprecated or leaving support?
    The question, as asked

    I'm preparing the technology risk section of the board pack. As of 28 September 2026, which business-critical and mission-critical applications in the Risk domain run on an IT component in either of these conditions? Use our EA repository. - Out of support: the component is rated deprecated and its end of support has already passed. - Leaving support: the component's end of support falls in the next 12 months, after 28 September 2026 and on or before 28 September 2027. Count only components an application actually runs on, not ones it has only evaluated, and go by the standardisation rating and the end-of-support date on each component's record. List only those applications, in one table with the application, its repository id, its business criticality, the component with its end-of-support date (the earliest that qualifies, if it has several), and the condition: "Out of support" or "Leaving support". If an application has components in both conditions, its condition is "Out of support".

    Asked at two scopes: risk domain and customer domain.

  • Platform standardisationWhich platform should be the standard, and what retires and moves?
    The question, as asked

    We're standardising our platforms, and I need the retirement list for the whole estate. Our rule: every platform serves one platform function (API management, analytics data, card processing and so on), each function keeps one standard platform, the one our EA repository rates standard, and every other platform of that function retires onto it. Platforms never consolidate across functions. Which platforms across the estate do we retire, and onto which standard? List one row per platform to retire, starting with that platform, in one table with the platform, its repository id, its function, its standardisation rating, the standard platform it retires onto, and the number of applications on it. Finish with the total number of applications that would have to move.

    Asked at two scopes: whole estate and corporate domain.

  • Migration wavesSequence the moves so no application moves before one it depends on
    The question, as asked

    We're retiring the Kubernetes Container Platform, each onto the standard platform of its own function: the platform our EA repository rates standard for the same platform function. Every application on a retiring platform moves to that standard. Most of those moves can happen in any order. Some can't: when one moving application depends on another (it consumes an interface the other provides, or the repository records that it requires the other), it has to move in a later wave than the application it depends on. Only dependencies between two moving applications count. Moving applications that depend on each other in a loop form a move group: they move together, in the same wave. List only the moving applications whose move is tied to another move in this way, in one table with the wave number, the application, its repository id and the standard platform it moves onto. Put each application in the earliest wave it can go.

    Asked at two scopes: kubernetes container platform and integration and data platforms.

3 questions

Risk and regulation

  • Outage impactIf this platform fails, which critical capabilities are hit?
    The question, as asked

    We are running an operational resilience scenario for the risk committee. If the API Gateway Platform went down for a day, what would the business lose? Which business capabilities would lose a business-critical or mission-critical application that runs on it? Use our EA repository. List only those capabilities, in one table with the capability and the critical applications on the API Gateway Platform that support it, each with its repository id. Use the capabilities the repository maps those applications to, without rolling them up to their parent capabilities.

    Asked at two scopes: api gateway platform outage and business intelligence platform outage.

  • ICT concentration riskWhich providers support several critical or important functions at once?
    The question, as asked

    For our DORA concentration-risk review I need the ICT third-party providers that more than one of our critical or important functions depend on, starting with the functions in the Payments domain. Work from our EA repository, with these rules: - A critical or important function is a business capability tagged "DORA critical or important function". Its sub-capabilities are part of it. - A function depends on a provider when the provider supplies an IT component that an application supporting the function runs on (not one it has only evaluated). An application supports the function when it is mapped to the function's capability or to one of its sub-capabilities. List only the providers that two or more of the Payments domain's critical or important functions depend on, in one table with the provider's name and repository id as the repository records them, the functions that depend on it, and the number of applications through which they do. Under the table, say in a sentence or two what this concentration means for our exit plans.

    Asked at two scopes: payments domain and risk domain.

  • Personal data transfersWhich interfaces send personal data outside the EEA without an adequacy decision?
    The question, as asked

    Our privacy team is mapping where personal data leaves the European Economic Area through our interfaces to a country with no adequacy decision we can rely on, starting with data sent from Payments applications. Use our EA repository, with these rules: - An interface sends data from its provider application to its consumer application. Count only that direction. - It carries personal data when its data classes include customer, contact or employee. - An application is in the EEA when its hosting location is in an EU member state, Iceland, Liechtenstein or Norway. - For this exercise, these countries outside the EEA have an adequacy decision: the United Kingdom, Switzerland, Japan, Canada, South Korea, New Zealand, Israel, Argentina and Uruguay. Every other country has none, including the United States, Singapore, India and Australia. List only the interfaces that send personal data from an EEA-hosted Payments application to an application hosted in a country with no adequacy decision, in one table with the interface, its repository id, the sending application and its hosting location, the receiving application and its hosting location, and the data classes.

    Asked at two scopes: payments domain and risk domain.

3 questions

AI governance

  • EU AI Act classificationWhich agents fall under a high-risk use case without human oversight?
    The question, as asked

    We're bringing our AI register into line with the EU AI Act, and the AI governance board wants the most urgent gaps first: agents that fall under a high-risk use case in Annex III and run with no human oversight at all. Please go through the agents on our three agent platforms (the agent hub, the cloud agent platform and the assistant agent platform). Classify each one from what it actually does: the class recorded on an agent is its owner's self-assessment and isn't always right. Use our reading of the Act for this register: - High-risk means an Annex III use case about natural persons: point 4, employment and worker management (recruiting and selecting candidates, decisions on promotion or termination, allocating tasks, monitoring or evaluating workers); point 5(a), eligibility for public assistance benefits and services; point 5(b), creditworthiness or credit scoring; point 5(c), risk assessment and pricing in life and health insurance. Detecting financial fraud is not 5(b), rating companies is not 5(b), and other insurance is not 5(c). - No human oversight means the agent's record names nobody who reviews its decisions, before or after they take effect. Pre-effect approval, sampling or aggregate review by a named person all count as oversight. List only the high-risk agents with no human oversight, in one table with the agent, its platform, its id, the Annex III point it falls under (4, 5(a), 5(b) or 5(c)) and what its record says about human review. Under the table, in a sentence or two, name any agent whose use would be a prohibited practice under Article 5.

    Asked at two scopes: all three agent platforms and agent hub and assistant agent platform.

  • AI inventory reconciliationWhich agents touching personal data have no record in the repository?
    The question, as asked

    We're reconciling our AI inventory with our EA repository before the AI register audit. I need the agents that touch customer or personal data but have no application record in the repository. The sweep covers the agents on our three agent platforms (the agent hub, the cloud agent platform and the assistant agent platform). An agent's record may refer to the repository application it belongs to, by fact sheet id or, where it gives no id, by name. An agent counts as registered only when that application exists in the repository; an agent with no reference, or with one the repository doesn't hold, is not registered. List only the unregistered agents that handle customer or personal data, in one table with the agent, its platform, its id, the customer or personal data it handles, and the repository reference on its record (or "none").

    Asked at two scopes: all three agent platforms and agent hub and cloud agent platform.

  • AI agent rationalisationWhich agents on different platforms do the same job?
    The question, as asked

    We're running an agent rationalisation review: where two teams have built the same agent on different platforms, we want to keep one of them. Which agents on our three agent platforms (the agent hub, the cloud agent platform and the assistant agent platform) do the same job as an agent on another of these platforms, under a different name? Two agents do the same job when they serve the same business process for the same kind of user and produce the same outcome; sharing a domain, a data set or an application isn't enough. The same agent registered in two inventories, and one product deployed under qualified names (EU, US, Pilot, V2, Deployment 3 and so on), don't count: we track those already. List only the duplicate pairs, one row per pair, in one table with the two agents, each one's platform and id, and the job they both do.

    Asked at two scopes: all three agent platforms and agent hub and cloud agent platform.

3 questions

Repository trust

  • Record completenessWhich critical applications can't be assessed yet, and what's missing?
    The question, as asked

    Before we start this year's application assessments in the Lending domain, I need to know which of our most critical applications we can't assess yet because our EA repository is missing the basics. Look only at the applications the repository rates mission critical or business critical. An application's record isn't complete when it has no owner, no review in the 12 months to 28 September 2026, no lifecycle phase, no annual cost, or a description that is just a code. List only the applications whose records aren't complete, in one table with the application, its repository id, its business criticality and what is missing. Under the table, say in a sentence or two which gaps you would get fixed first and by whom.

    Asked at two scopes: lending domain and platform domain.

  • Repository and CMDB reconciliationWhere do the repository and the CMDB disagree on owner, criticality or cost?
    The question, as asked

    Before the APM data-quality review I want to know where our EA repository and our CMDB disagree about the applications in the Risk domain. First pair each repository application with its CMDB business application: a CMDB record that carries a repository id belongs to that application; one that carries none belongs to the repository application whose name or alias equals its name once both are lower-cased, a trailing environment marker (PRD, PROD or Production, with or without a dash, underscore or brackets) is dropped and punctuation is read as a space. About a third of CMDB records carry no repository id, so most of the work is in that second step. A record on either side that pairs with nothing is an orphan, not a disagreement. Then compare each pair's owner, business criticality and annual cost. The CMDB grades criticality from 1 to 4; read those as the repository's four levels in the same order (1 is mission critical, 4 is administrative service). A value that is blank in either system is a gap, not a disagreement, so leave it out. List only the applications where the two systems disagree, in one table with the application, its repository id, the CMDB business application it pairs with (its name and number), the field they disagree on, the repository's value and the CMDB's value. Under the table, say in a sentence or two how you would get each disagreement resolved.

    Asked at two scopes: risk domain and payments domain.

  • Retired but still runningWhich applications marked retired still have live infrastructure or incidents?
    The question, as asked

    Our EA repository says a number of applications in the Lending domain are at end of life, and I suspect some of them are still running. Which applications does the repository mark end of life (lifecycle phase end of life) whose CMDB service still shows signs of life, as of 28 September 2026? Use our EA repository and our CMDB, with these rules: - Pair each application with its CMDB business application and that business application's application service: a CMDB record that carries a repository id belongs to that application; one that carries none belongs to the repository application whose name or alias equals its name once both are lower-cased, a trailing environment marker (PRD, PROD or Production, with or without a dash, underscore or brackets) is dropped and punctuation is read as a space. - A sign of life is a server of that service that is still installed and operational, or an incident on the service, or a change on the service or one of its servers, opened in the 90 days to 28 September 2026 (from 30 June 2026). - The change that decommissions the application (its name starts with "Decommission") is not a sign of life. List only those applications, in one table with the application, its repository id, its CMDB service, and the evidence that it still runs: the operational servers, or the incident or change numbers.

    Asked at two scopes: lending domain and whole estate.

Next edition

Coming next

Questions that need an architect's judgement: design review against standards, carve-out scope, transition architecture and architecture decisions.

Get the next edition by email with the kit.

One question, end to end

AI inventory reconciliation

The question, as asked

“We're reconciling our AI inventory with our EA repository before the AI register audit. I need the agents that touch customer or personal data but have no application record in the repository. The sweep covers the agents on our three agent platforms (the agent hub, the cloud agent platform and the assistant agent platform). An agent's record may refer to the repository application it belongs to, by fact sheet id or, where it gives no id, by name. An agent counts as registered only when that application exists in the repository; an agent with no reference, or with one the repository doesn't hold, is not registered. List only the unregistered agents that handle customer or personal data, in one table with the agent, its platform, its id, the customer or personal data it handles, and the repository reference on its record (or "none").”

What a right answer contains
  • The 6 unregistered agents that handle customer or personal data, each with its platform and the repository reference on its record
  • None of the 3 unregistered agents that touch no personal data
  • None of the 3 agents that name their application without an id but are registered, because the repository holds that application

Listing any of those 6 look-alikes, or an agent that doesn't exist, fails the answer.

How we do it

The method, in brief

A realistic estate

  • 1,200 applications in six domains of a retail bank, with capabilities, technology, providers, interfaces and data
  • A CMDB, a delivery tracker and three AI agent platforms, served as simulated systems modelled on public API documentation
  • Messy where real estates are messy: missing owners, stale reviews, records that don't match across systems

Answers checked by code

  • Every answer key is computed from the records
  • Complete and correct lists score; a trap or an invented record fails the answer
  • Three runs per question, so we can report consistency

Published, so you can check

  • Every question as asked, with what a right answer contains and the traps that should fail it
  • The scoring rules, in plain language

We found no public benchmark of estate-scale EA analysis. Tell us if we missed one: support@aitransformstudio.com

The question kit

Run the 17 questions on the AI you use today

Each question as asked, what a right answer contains, the traps that should fail it, and how to score it. Free, with the results and the next edition by email when they're out.

Read the kit now

One email to confirm, then the results and the next edition when they're out. Unsubscribe any time. How we use your email.

Questions

About EA-Bench

What is EA-Bench?

EA-Bench is a study of how well AI systems answer the analysis questions enterprise architecture teams get, on a realistic estate where the right answers are known. AI Transform Studio publishes it, with the questions and the scoring rules open for anyone to check. Edition 1 covers estate analysis: 17 questions in five areas.

Can I run the questions on my own AI assistant?

Yes. The question kit has each question as asked, what a right answer contains, the traps that should fail it, and the scoring rules. Run them on any assistant connected to your systems, and score the answers against records you can check.

Can I run them on my own estate?

Yes. In AI Transform Studio, the same questions run against your connected EA repository, CMDB and delivery tracker, and every figure in the answer shows the record it came from.

When are the results out?

We haven't published results yet. Ask for the question kit with your email, and we'll send you the results when they're out.

Who made EA-Bench, and is it independent?

AI Transform Studio made it, and Studio is one of the systems we test. That is why the questions and the scoring rules are published: you can rerun any question and check the answer yourself.

Why aren't EA tools' built-in assistants tested?

Their licences usually forbid publishing benchmark results. The kit lets you test yours yourself.

Ask your own estate the same questions

Connect your EA repository and service management, ask the questions you already get, and see which records each figure came from.

Bring your own model keys.

EA-Bench: an AI benchmark for EA teams · AI Transform Studio