Many AI projects do not fail because of the technology, but because of an unclear start. A team tries several tools at the same time, measures no baseline and after a few weeks can prove neither benefit nor risk. A good AI deployment plan within the team reverses this order: First a concrete problem is described, then a limited pilot is set up and only after reliable results is a decision made about the rollout.
The following seven-step plan is intended for Austrian companies that want to test AI in practice without immediately reorganizing an entire area. It separates pilot and production, protects real data and makes both success and termination measurable. Legal obligations depend on the specific system and purpose of use; for sensitive or consequential applications, the responsible specialist authorities should be involved early.
Why a focused pilot is better than a large AI initiative
"We have to do something with AI" sounds like a fresh start, but it does not provide a testable goal. A limited use case is much stronger: "We test whether an approved assistant produces usable drafts for internal knowledge questions from public product documentation." This allows the data, user group, quality standard and expected benefit to be defined.
The WKO recommends in its current AI tips for SMEs, to clarify the status quo and goals and to start with pilot projects in a limited area before rolling out company-wide. That's exactly what the deployment plan is for: it turns a vague idea into a controlled experiment with a real decision at the end.
Step 1: Select a problem with a baseline
The pilot doesn't start with the tool, but with a recurring task. Suitable are processes that occur frequently enough, have a clear output, and don't cause immediate harm during testing. Examples include initial outlines, internal summaries of approved documents, categorization of non-critical requests, or drafts for learning materials.
Before the first AI test, the current process is measured. How long does the task take? How many corrections are needed? Which errors occur? How satisfied are the participants? A small sample of ten to twenty real cases, prepared in a privacy-compliant way, is often enough for a baseline. Without a baseline, any later time savings become estimates.
A good problem statement contains four elements:
- the specific task,
- the current burden or quality gap,
- the affected user group and
- the boundary of what should not be automated.
Example: "The support team takes an average of twelve minutes to draft an internal reply. The pilot should reduce drafting time without transferring customer data to an unapproved system and without sending replies automatically."
Step 2: Define purpose, risk and stakeholders
A general language model can perform many tasks. For the assessment, however, the specific use matters. The RTR overview of risk levels for AI systems explains the risk-based approach of the AI Act and notes that classification and obligations depend on the intended use. A draft text for internal training should be treated differently than a system that evaluates applicants.
The pilot sheet should therefore record:
- What may the system do during the test?
- Which decision remains entirely with a human?
- Which types of data are allowed, anonymized, or excluded?
- Who is affected by the outcome?
- Will content be used internally or visible externally?
- Which teams must approve or be consulted?
For employee data, recruiting, performance evaluation, or monitoring, HR, data protection, and, where applicable, the works council are not a later checkpoint but part of the planning. Prohibited practices or potentially high‑risk applications should not be included in an improvised team test.
Step 3: Check the tool and data flow before testing
Only now are potential tools compared. Functionality is only one criterion. Equally important are the contract, storage locations, access controls, logging, deletion options, use of inputs for training, subcontractors, and changes to terms. A free service may suffice for public idea collection and be completely unsuitable for internal records.
Draw the data flow on a single page: Where does the input come from? Who prepares it? To which service is it transmitted? Where is the output stored? Who can access it? When is it deleted? This sketch often uncovers open questions that remain invisible in a product presentation.
Data minimization applies to the pilot. Instead of using complete customer files, synthetic or consistently anonymized test cases are preferred. If real data appears to be strictly necessary, it must be precisely justified and checked to determine why. The WKO describes in its "notes on customer-related data, that the legal basis, confidentiality and the information provided to data subjects must be carefully observed.
Step 4: Prepare test cases and quality criteria
An AI pilot is not free-form experimentation. The team draws up a test catalog in advance that contains easy, typical, and difficult cases. Known problematic cases are also included: ambiguous formulations, outdated information, dialect, contradictory information, or inputs that could lead the system to impermissible conclusions.
An expected quality is described for each test case. For a draft response, the criteria can include factual accuracy, completeness, clear language, an appropriate tone, sources, and freedom from personal data. A simple scale from zero to two per criterion makes results comparable:
- 0: unusable or risky,
- 1: usable with substantial rework,
- 2: nach einem standardmäßigen Experten-Review nutzbar.
Der Referenzwert entsteht aus menschlich erstellten Ergebnissen derselben Fälle. Wichtig ist nicht, ob die KI 'kreativ' wirkt, sondern ob der gesamte Prozess einschließlich Kontrolle besser wird.
Schritt 5: Den Pilot mit klaren Rollen durchführen
Ein kleiner Pilot braucht mindestens vier Verantwortlichkeiten, die in einem SME auch von zwei Personen abgedeckt werden können:
- Pilotverantwortung: hält Umfang, Termine und Entscheidungen zusammen.
- Fachreview: bewertet Inhalte anhand der vereinbarten Kriterien.
- Tool and data protection review: monitors configuration, access, and data flow.
- Application: processes test cases and documents effort and anomalies.
The test runs within a defined period, for example four weeks. The number of users remains small. No one may quietly expand the pilot into a production process. External communications, personnel decisions, or contractual consequences are not triggered automatically.
If common ground rules are still missing, the jobspot.at guide to AI policy in the company. For daily task distribution, it is supplemented by the AI checklist for managers. The pilot does not have to reinvent these decisions, but should apply them concretely to its use case.
Before the start, all participants receive a brief, application-specific training. Article 4 of AI Act stellt bei KI-Kompetenz unter anderem auf technisches Wissen, Erfahrung und Einsatzkontext ab. Für den Pilot heißt das praktisch: Das Team kennt Grenzen des Systems, erlaubte Daten, Qualitätskriterien, Eskalationsweg und Stopprecht.
Step 6: Jointly evaluate benefits, errors and workload
After each test case, a few but meaningful metrics are recorded:
- Time for preparation, AI use, and follow-up review,
- Quality points per criterion,
- Number and type of critical errors,
- Share of completely discarded outputs,
- Feedback from users,
- Incidents or near-misses involving data and security.
Post-processing time is particularly important. A draft in ten seconds saves nothing if an expert then needs twenty minutes for source and fact corrections. Likewise, a small time saving can still make sense if quality and accessibility increase significantly.
The evaluation separates systematic from random errors. Repeatedly fabricated sources, disadvantaging certain groups, or uncontrolled data leaks are not optimization details. They can be grounds for termination. The Austrian Praxisleitfaden Digitale Verwaltung und Ethikshows with a decision tree and an ethical checklist how technical opportunities can be linked with transparency, protection of fundamental rights, and human-centered design. These principles are also useful review questions outside of the administration.
Step 7: Decide Go, Adapt, or Stop
In the end there is a deliberate decision, not a creeping permanent pilot. Three outcomes are possible:
Go with a controlled rollout
The pilot meets minimum quality, utility and protection requirements. For the rollout, user groups, training, support, access rights, review cadence and the responsible party are defined. The production process will continue to receive quality controls and a monitoring date.
Adjust and retest
The benefit is apparent, but individual criteria are missed. In that case exactly one key variable is changed, for example data preparation, prompt template, model, expert review or deployment boundary. This is followed by a new limited test. If multiple things are changed at the same time, the effect can no longer be attributed.
Stop
The process is terminated if risks cannot be controlled, results remain unreliable, or the effort required for controls outweighs the benefits. A stop is not a failure of innovation. It protects resources and provides knowledge for a better-suited use case.
Record the decision on one page: verified purpose, test scope, key metrics, identified risks, conditions and responsible approval. For a go, also include which assumptions must continue to hold. If, for example, the model, the data path or the user group changes, the previous approval is not automatically sufficient. For an adjustment, document which problem the change is intended to solve. For a stop, access rights, test data and temporary integrations are removed in a controlled manner. This ends the pilot cleanly both technically and organizationally, and a later team does not have to answer the same questions from scratch.
The compact four-week plan
Week 1: Problem and scope
Measure the baseline, define objectives and non-goals, identify stakeholders, check data classes and perform an initial risk screening. The result is an approved pilot sheet.
Week 2: Preparation
Check tools, configure secure access, prepare test data, create an evaluation scheme and train the small pilot team. There is still no production use.
Week 3: Controlled test
The prepared cases are processed. Time, quality, errors and feedback are documented immediately. Critical deviations stop the test until resolved.
Week 4: Decision
Compare results with the baseline, consult affected parties, assess outstanding risks and decide on go-ahead, adjustment or stop. The decision includes responsible parties, conditions and the next review date.
Practical example: Review internal job postings
An Austrian company wants to word job advertisements more clearly and inclusively. The pilot is to flag existing, already approved postings for hard-to-understand sentences and unclear requirements. It must not analyze applications or make selection decisions.
The team first measures the previous processing time and defines criteria: facts must not be changed, mandatory and optional requirements must not be swapped, and no new benefits may be promised. Twenty anonymized ads form the test catalogue. HR checks each output, documents corrections, and compares it with a human revision.
After four weeks the pure drafting time decreases, but the system occasionally removes technically necessary terms. The pilot is not rolled out immediately. Instead it receives a stricter template and a mandatory review of the requirement lists. This result is valuable because it reveals a concrete weakness before real job ads are published without review.
Seven common pilot mistakes
- The team chooses a tool before the problem is defined.
- The pilot uses real sensitive data, even though test data would suffice.
- There is no baseline and therefore no demonstrable benefit.
- Only simple cases are tested; edge cases remain invisible.
- Follow-up checks and training time are missing from the cost accounting.
- A test output already triggers production decisions.
- The pilot has no end date and no stopping criteria.
Frequently asked questions about the AI deployment plan
How large should the pilot team be?
As small as possible and as diverse as necessary. Often three to six people from the business unit, application, and control functions are sufficient. What matters are representative cases and clear roles, not a large number of participants.
Can a pilot start with a free tool?
Only if the contract, data processing, and intended purpose are appropriate. This may be possible for public, artificially generated test data. Confidential or personal information does not automatically belong in a free offering.
When is an AI application high-risk?
That can't be inferred from the product name alone. Purpose, modes of use, and the areas listed in the AI Act are decisive. RTR provides overviews on this; if in doubt, the specific case should be assessed by qualified review.
What remains after a successful pilot?
A production deployment plan with responsible owners, an approved purpose, training, data rules, quality gates, support, an incident response procedure, and a review date. A pilot report alone is not yet a safe operation.
Conclusion: First validate, then scale
A good AI deployment plan keeps the test small and the decision large. It connects a real work problem with a baseline, a risk screen, secure data, representative test cases, and measurable quality. That way a team sees not only what the tool produces, but whether the whole workflow improves and can be operated responsibly.
For the start, choose a frequent, clearly delimited task that has no automatic consequences for people. Measure ten cases in the current process and then define success and termination criteria. This first step is unspectacular, but it separates a robust pilot from an expensive product demo.
