Goal-driven marketing agents under explicit constraints: a design

1Context and motivation

Marketing automation in commerce has moved from manual campaigns to rule-based journeys: schedules, triggers and spend splits written in advance. Rules are legible, but each one encodes a single moment's view of what works. When the relative performance of channels shifts, or when a merchant sells into several regions with different consent rules and channel habits, the rules have to be rewritten by hand, and nothing in them weighs one objective against another.

We describe a design for goal-driven marketing agents: systems that choose budget allocation, channels, audiences, creative and timing toward an objective that an operator states, such as contribution margin or a lifetime-value proxy, inside constraints the operator also states. "Agent" here means a learning decision policy in the sense used in reinforcement learning, which maps observed states to actions so as to maximise a cumulative reward¹. It does not mean a system that acts unchecked: in our design people set the goals and the limits, approve high-impact actions, and can stop the agent at any time.

This paper describes the architecture and the reasons for each part of it. It does not report effects, because we have not measured any.

2Levels of automation

Automation is not a single switch. One widely used model separates four classes of function (information acquisition, information analysis, decision and action selection, and action implementation), each of which can be automated at a level anywhere from manual to automatic, and it stresses that automation changes the operator's work rather than removing it².

We use that model to place the design. Analysis and decision selection are automated at a high level for low-impact, reversible choices, such as which of two approved subject lines a customer receives. They are automated at a lower level, with operator approval, for high-impact choices, such as moving a large share of a budget to a new channel or deepening a discount. The operator's work shifts toward specifying goals, constraints and evaluation. The interface follows published guidelines for human-AI interaction: make clear what the system can do, support efficient correction and dismissal, and make clear why the system did what it did³.

3Problem formulation

The agent observes a state, takes actions, receives rewards, and must respect constraints.

• State. Customer: lifecycle stage, engagement history, consent and channel preferences, region. Merchant: budget, stock, margins, brand rules, active markets. Channel: recent cost and response by channel. Context: season, holidays, platform policy changes.
• Actions. Budget allocation across channels and campaigns; campaign start, stop, bids and audiences; creative and offer choice within approved sets; send timing and frequency.
• Rewards. Short-term terms (orders, contribution margin) and long-term terms (retention, a lifetime-value proxy), net of media and discount cost.
• Constraints. Consent and contact rules per jurisdiction, contact frequency caps, discount depth ceilings, brand rules, prohibited targeting attributes, and minimum exposure for cohorts where a skew would cause harm.

We formulate the allocation problem as a constrained Markov decision process: maximise expected cumulative reward subject to bounds on expected cumulative costs⁴. We keep constraints separate from the reward, rather than folding them in as penalty weights, for two reasons. First, a constraint such as a legal contact rule is not something to trade against revenue at any price. Second, collapsing several objectives into one with fixed weights is impossible, infeasible or undesirable in identifiable situations, for example when the weights are unknown in advance or change over time⁵. Where the trade-off between margin and retention is genuinely a matter of choice, the operator sets it explicitly and the setting is versioned.

4Architecture

The design has four layers: state and data, decision modules, a policy engine, and an explanation, review and audit layer.

4.1State and data

Events from storefronts, channels and ad accounts are joined into the state described above. Each record carries consent scope and region, and a feature is only available to a decision whose purpose that consent covers. This is a design rule, stated here because every later layer depends on it.

4.2Decision modules

Three kinds of decision run at three time scales.

• Message-level choices (which creative, which offer, send or hold) run as contextual bandits. A contextual bandit selects among actions using context about the user and the item, and adapts from observed feedback; logged data from randomised traffic can also be used to evaluate a bandit policy offline⁶.
• Budget and channel allocation over weeks runs as reinforcement learning. A greedy, one-step learner treats two visits from the same customer as unrelated, and so optimises the next click; a reinforcement-learning policy can optimise value over several steps, which is closer to lifetime value⁷.
• The learning signal is incremental effect. Rewards are credited against randomised holdouts, and response models estimate the change in outcome caused by an action rather than the outcome itself, which is the uplift-modelling formulation⁸. We do not rely on observational comparisons: in large advertising field experiments, common observational methods often failed to recover the effect measured by randomisation⁹.

Deployed reinforcement learning faces problems that benchmark settings hide, among them limited samples, safety constraints, delayed rewards, partial observability and the need for explainable policies¹⁰. The design answers these with conservative exploration, hard constraints, holdouts, and a review layer, and it uses reinforcement learning only for the slow, budget-level decisions where its multi-step view matters.

4.3Policy engine

Before any learned policy is consulted, the policy engine removes infeasible actions: contacts without the required consent, sends above the frequency cap, discounts beyond the ceiling, audiences built on prohibited attributes, and creative outside the brand rules. Constraints are written as versioned rules owned by people. The agent cannot edit them.

4.4Explanation, review and audit

Every decision is logged with the state summary, the candidate actions, the constraint checks that removed actions, the chosen action, the policy version and a short rationale. Decisions above an impact threshold (share of budget moved, discount depth, size of audience) wait in a review queue for operator approval. Operators can pause the agent per campaign, region or channel.

Table 1 maps each component to the problem it addresses and the published work it rests on.

Component	Problem it addresses	Evidence
Levels of automation per decision type	Uniform automation either overloads operators or removes needed oversight	2
Constraints held apart from the reward	Fixed weights cannot express hard limits or unknown trade-offs	4, 5
Contextual bandits for message choices	Static rules cannot adapt to feedback	6
Reinforcement learning for budget allocation	Greedy learners optimise the next click, not lifetime value	7
Uplift as the learning signal, with holdouts	Observational comparisons misstate advertising effects	8, 9
Doubly robust offline evaluation	New policies cannot be tested on live customers first	11
Value-based reward with side-effect checks	Misspecified rewards produce unwanted behaviour	12
Exposure monitoring by cohort	Cost-optimised delivery can skew who is reached	13, 14
Review queue, explanations and override	Operators cannot correct what they cannot see	3

Source: our design, with the works cited in the table.

5Reward design

The choice of reward shapes behaviour more directly than the choice of algorithm. Work on accident risk in machine learning names two failure modes that come from a wrong objective: negative side effects, and reward hacking, where a system finds a way to score well that its designers did not intend¹². In marketing, a reward based on clicks or opens is the obvious case. Table 2 shows the arithmetic in its simplest form.

Quantity	Action A: broad discount banner	Action B: targeted replenishment message
Click probability per impression (input)	0.10	0.04
Contribution margin per click, currency units (input)	2.00	8.00
Expected margin per impression (computed: probability times margin)	0.20	0.32
Chosen by a click reward	Yes	No
Chosen by a margin reward	No	Yes

Source: Worked model; inputs in the table.

A click reward picks Action A because 0.10 is greater than 0.04. A margin reward picks Action B because 0.32 is greater than 0.20. A margin reward is still incomplete: it cannot see a later cost of an action, such as a discount that shifts purchases from a future week into this one. The design therefore adds a lifetime-value proxy with a long horizon, and it tracks side-effect metrics (unsubscribes, complaints, average discount depth) as guardrails that can halt a policy, rather than as terms the agent may trade away.

6Regional adaptation

Region enters the design in two separate ways. Legal and consent rules per jurisdiction are hard constraints in the policy engine; they are inputs written by people, not patterns the agent learns. Behavioural differences between regions, such as which channels respond, are learned: region is part of the state, models share parameters across regions, and a region with little data borrows from the others until its own data accumulates. We make no claim here about how any region behaves; that is what the evaluation below would have to establish.

7Evaluation protocol

Before any claim about outcomes could be made, a policy would pass three stages.

1. Offline evaluation. New policies are scored on logged decisions with doubly robust estimators, which give accurate value estimates when either the reward model or the model of the logging policy is good¹¹. This requires that the logging policy record the probability of each action it took, so the design logs propensities from the start.
2. Randomised holdouts online. Each campaign keeps a randomised holdout, and outcomes are reported as incremental effects with confidence intervals. Sample sizes are set in advance, because advertising effects are small relative to the noise in individual sales: in 25 large field experiments with retailers and brokerages, the median confidence interval on return on investment was over 100 percentage points wide¹⁵.
3. Staged rollout. A policy runs in shadow mode first, then on a small share of traffic, widening only while guardrails hold.

The primary metric is incremental contribution margin per unit of spend against the holdout, with retention at a fixed horizon as a secondary metric. Guardrails are unsubscribe and complaint rates, average discount depth, policy-engine rejections, and exposure share by cohort and region.

8Fairness, privacy and oversight

An optimiser can produce skewed outcomes without anyone intending them. In a field test, an ad for STEM careers that was meant to be gender neutral was shown to fewer women than men, because the delivery algorithm optimised cost-effectiveness and younger women were more expensive to reach¹³. A study of a large ad platform found significant skew along gender and racial lines in the delivery of real employment and housing ads despite neutral targeting, driven by budget, ad content and the platform's relevance predictions¹⁴. A marketing agent that allocates spend by expected return is subject to the same mechanism. The design therefore monitors exposure by cohort and region, applies minimum-exposure constraints where a skew would cause harm, and reviews features that may act as proxies for protected attributes.

On privacy, the agent works only with consent-scoped data, does not infer sensitive attributes, and treats opt-outs and channel preferences as hard constraints. On manipulation, the policy engine excludes tactics we judge to exploit vulnerability, such as false urgency, whatever their measured response. These are design choices, not findings.

9Limitations

• No measurements. We have not measured the effect of this design on revenue, retention, fairness or brand safety. Every benefit implied above is a hypothesis for the protocol in the evaluation section.
• Reward proxies. The lifetime-value proxy is itself a model, and errors in it become errors in the agent's goal.
• Exploration cost. Learning requires trying actions that may turn out worse; with small merchants and small budgets, holdouts and exploration consume a real share of the budget, and estimates stay noisy for a long time.
• Constraint coverage. The policy engine enforces only the rules someone has written. A harmful behaviour that no rule anticipates is caught only by monitoring, if at all.
• Interaction with platforms. Ad platforms run their own optimisers. The agent's actions pass through them, so the agent controls inputs, not delivery.
• Operator effects. Outcomes depend on how operators set goals and use the review queue, a human-factors question this design does not answer.

10Conclusion

We have described a design for goal-driven marketing agents framed as constrained sequential decision-making: bandits for message choices, reinforcement learning for budget allocation, incremental effect as the learning signal, hard constraints in a policy engine people own, and a review layer that keeps high-impact decisions with operators. Each part rests on published work summarised in Table 1. What the design lacks is evidence of its own effect; the next step is to run the evaluation protocol on a single decision type, message choice, before widening to budget allocation.