Multi-Agent Calendar Assistant: A Real-World Case Study

Calendar management looks like an easy job for an AI agent until you test it. Consider a request like "move tomorrow's design review to Thursday afternoon." The system has to find the right event, check what Thursday afternoon already holds, and edit one record without touching a similar one. ScheduleMe is a research system built to handle requests like this through conversation. Researchers at the University of Moratuwa built it, with Srinath Perera of WSO2 as a co-author, and the paper appeared at PACLIC 39. This case study covers the architecture, the scaling extension, both evaluations, and the failure modes the authors documented. The results below come from the paper, including the ones that fell short.

What ScheduleMe Is and Why It's a Useful Multi-Agent Example​

ScheduleMe manages Google Calendar events through natural-language conversation. A supervisor agent talks to the user and hands each calendar operation to one of four specialist agents. The paper is "ScheduleMe: Multi-Agent Calendar Assistant." Four of its authors are from the University of Moratuwa: Oshadha Wijerathne, Amandi Nimasha, Dushan Fernando, and Nisansa de Silva. The fifth is Srinath Perera, who leads WSO2's research team. It is published in the PACLIC 39 proceedings and on arXiv.

ScheduleMe is not a WSO2 product. It runs on an open-source Python stack with OpenAI's GPT-4o mini for reasoning. We present it here as a worked example of multi-agent design tradeoffs.

The calendar domain suits that purpose for three reasons:

• Every operation has a checkable outcome. An event was created with the right title and time, or it was not. That makes task success measurable.
• The operations have real side effects. Editing and deleting change a user's data, so routing mistakes have visible costs.
• The inputs are ambiguous in familiar ways. Relative dates, similar event titles, and multilingual phrasing all stress the coordination layer.

The paper positions ScheduleMe against traditional task-oriented dialogue systems. Those systems rely on intent classification and slot filling, usually through rigid, frame-based flows. The authors note that such flows struggle when users give details out of order, revise a command, or phrase things unexpectedly. ScheduleMe replaces the fixed flow with an LLM supervisor that asks follow-up questions until it has what each operation needs.

The Architecture: One Supervisor, Four Specialists​

The supervisor is the only agent that talks to the user. It reads each message, works out the intent, and collects the required details before delegating. The four specialists handle scheduling, availability checking, event editing, and event deletion. All communication between agents passes through the supervisor. When a specialist needs more information, it asks the supervisor to go back to the user.

Figure 1: ScheduleMe's supervisor-and-specialists architecture, based on Wijerathne et al. (2025), Figure 1 and Section 3.2.

Each specialist follows the ReAct pattern, which interleaves reasoning steps with tool calls. Its tools are custom functions that wrap the Google Calendar API. The appendix prompts name them directly. The scheduling agent uses check_calendar_conflicts and create_calendar_event, and the availability agent calls check_availability(start_date, end_date). The editing agent uses update_event, and the deletion agent calls delete_event with an event ID.

The specialists are stateless. Each one receives a structured payload, runs its task, and returns the result to the supervisor. The supervisor then reports the outcome to the user. This is the supervisor/specialist orchestration pattern in practice: one stateful coordinator and several narrow, stateless workers.

The stack​

The implementation is small enough to list in full:

Two traffic paths are worth keeping apart when reading this design. User requests travel inward from the Streamlit UI through FastAPI to the agent graph. Model calls travel outward from the agents to GPT-4o mini through LangChain. Calendar operations are a third path, from each specialist's tools to the Google Calendar API.

The routing contract​

The supervisor prompt requires a JSON reply with two fields. next names the agent to call, the user, or FINISH. messages holds the instruction or the reply. An illustrative reply in that format looks like this:

{
  "next": "event_scheduler_agent",
  "messages": "Schedule an event titled 'Design review' on 2025-05-01 at 10:00 AM."
}

The graph then routes on next. Two details in the published prompts stand out for anyone building a similar system.

First, some safety rules live in prompts rather than code. The scheduling prompt says the agent must check for conflicts before creating an event. That works only while the model follows the instruction. Moving the same rule into the tool layer would make it enforceable. Even an enforced check is not a reservation, though. Two concurrent requests can both see a free slot before either one writes. Production scheduling needs concurrency control at the calendar itself, such as a reservation or a conditional update. It also needs idempotent event creation, so that a retried call does not create a duplicate. This is an engineering inference from the check-then-write flow. The paper did not test for this race.

Second, the agent names in the supervisor prompt are not fully consistent. The agent list includes event_modifier_agent and event_remover_agent. The allowed values for next include event_editor_agent instead and omit the deletion agent. The paper does not say whether the running code shared this mismatch. Still, it shows how easily a prompt and a routing graph can drift apart. Generating both from one definition avoids that class of bug.

Scaling Past a Single Supervisor​

The base design serves one user well. The authors note that real deployments need concurrency, so they added a distributed supervisor architecture. Its components are:

• Parallel supervisors: Multiple supervisor instances run at once, each with a unique identifier.
• A session-aware load balancer: A custom balancer routes each session to the least-loaded supervisor. It keeps session affinity and reassigns sessions when an instance fails.
• Shared state in Redis: Redis holds session context and state metadata for all supervisors. Its time-to-live (TTL) setting clears inactive sessions automatically.
• A dynamic agent registry: Agents can be discovered, activated, and deactivated at runtime. Tasks go out through a thread pool executor, with capability-based routing.
• An asynchronous backend: The backend uses async/await so that concurrent conversations do not block the event loop.
• Containerized deployment: Docker Compose runs multiple instances against the same Redis backend. An Nginx reverse proxy provides round-robin HTTP load balancing, health checks, and SSL termination.
• A metrics endpoint: It reports active sessions, supervisor load, and agent utilization.

That last item is the metrics and monitoring layer added for the distributed version. It covers system load rather than agent quality, which the experiments measured separately.

A few points matter when reading this section as an architect. The paper describes two balancing mechanisms. One works at the session level, picking the least-loaded supervisor. The other is Nginx round-robin at the HTTP level. The paper reports both mechanisms but does not describe how they compose, so the relationship between them is unclear.

Shared state in Redis is what allows a session to move to another supervisor after a failure. It also makes Redis a dependency that every supervisor relies on. The authors say the design removes the single point of failure at the supervisor tier. The paper does not discuss Redis replication, so that guarantee depends on how Redis itself is deployed.

Finally, the paper reports no throughput or latency measurements for the distributed version. Its experiments evaluate task success and usability. The scaling design is an architecture description, not a benchmarked result.

How It Was Tested: Two Real Experiments​

Direct comparison with earlier scheduling systems was not practical. The authors explain that most traditional assistants are rule-based and lack public evaluation datasets. They also skipped head-to-head tests with commercial assistants, because those APIs are closed and their task coverage differs. Instead they ran two experiments, and the way they measured task success and usability is as useful as the results.

Experiment 1: zero-shot multilingual commands​

The team tested six languages: English, French, German, Tamil, Sinhala, and Chinese. Each language had 20 test inputs, with 5 for each of the four task types. That gives 120 test cases in total. No fine-tuning was done. An output counted as correct only if the assistant read the intent correctly and executed the calendar action with the correct parameters.

Source: Wijerathne et al. (2025), Table 1.

The column pattern says more than the totals. Scheduling held up in every language except Chinese. Editing and deletion degraded most, and those are the operations that change or remove existing data. The authors observe that, apart from Chinese, the ranking follows the language resource levels proposed in earlier work by Ranathunga and de Silva.

Two caveats apply. Each cell holds only five cases, so one failure moves a cell by 20 percentage points. The authors also note they are native Sinhala and English speakers. Outputs in the other four languages were not checked by native speakers, so those error rates may be off in either direction.

Experiment 2: a 20-person user study​

Twenty active digital calendar users worked with ScheduleMe through a web interface linked to test Google Calendar accounts. Each participant completed 5 to 7 operations, mixing simple, complex, and multilingual requests. Participants self-reported task success and errors. They then completed a System Usability Scale (SUS) questionnaire and rated trust and satisfaction on five-point Likert scales.

Source: Wijerathne et al. (2025), Table 2.

Figure 2: An illustrative conversation showing how the supervisor routes a rescheduling request. It is not a transcript from the study. The availability check and the edit are separate steps, so the slot is not reserved between them.

Raw numbers need a reference point. MeasuringU's analysis of nearly 1,200 usability tasks puts the average task-completion rate at 78%, with top-quartile tasks above 92%. For SUS, Bangor, Kortum, and Miller report a mean of 68.2 for web interfaces. In their adjective-rating data, "Good" corresponds to a mean SUS of 71.4 and "Excellent" to 85.5. ScheduleMe's 82.5 sits above the web mean and closer to "Excellent" than to "Good."

These comparisons are indicative rather than exact. ScheduleMe's success rate was self-reported, the sample was small, and the researchers designed the tasks. The benchmark datasets come mostly from observed usability tests of commercial software. The two benchmarks also measure different things. SUS reflects perceived usability, while the task-success figure reflects participants' own reports of completion.

What Went Wrong: Failure Modes Worth Knowing​

The paper's error analysis names three failure modes. Each one points to a specific design decision.

Translation-induced errors​

In multilingual sessions, some non-English event titles were translated or normalized to English internally. Follow-up queries then failed to match the stored title, which led to wrong retrievals or deletions. A title that the model rewrites stops working as an identifier. The deletion prompt already requires an event ID. The likely weak point is the step before it, where the system resolves a typed title to that ID.

Task parsing errors​

Complex date and time expressions or ambiguous phrasing caused two opposite failures. Sometimes the system asked for too many clarifications. Sometimes it sent the task to the wrong specialist. The supervisor prompt tells the model to ask the user rather than guess whenever something is unclear. That rule protects against wrong actions, but it pushes ambiguous inputs toward over-clarification.

Entity reference confusion​

When several events had similar titles, the system sometimes picked the wrong one to edit or delete. The limitations section says that relying on titles for disambiguation, without persistent conversational memory, increases the risk of this error.

All three errors were more frequent in non-Latin scripts: Tamil, Sinhala, and Chinese. The authors attribute this to semantic drift during translation and limited multilingual training coverage. The pattern matches Table 1, where the edit and delete columns show the largest drops.

For mitigation, the authors plan multilingual entity handling, context tracking, and confidence-based fallback. Their future-work list adds self-consistency with majority voting and constrained decoding with schema or rule checks. It also includes guarded tool calls with retries and backoff, plus escalation pathways. These target the operations where a wrong answer deletes or changes something.

Limitations and What the Researchers Would Do Differently​

The paper's limitations section covers memory, infrastructure, evaluation scope, and privacy. The table pairs each stated limitation with the direction the authors propose.

The memory row connects to the failure analysis, but the paper's evidence on memory is mixed. Sections 3.2 and 3.3 describe a persistent JSON state file and Redis-backed session context for multi-turn interactions. Section 8 still lists the lack of persistent conversational memory as a limitation. The paper does not define exactly where within-session state ends and durable conversational memory begins. It says the gap increases the risk of errors when events share similar names, but it does not measure how many of the documented failures the gap caused.

The privacy discussion deserves a closer read. ScheduleMe sends event titles and notes, locations, participant names, and times to cloud-hosted LLMs and Google Calendar APIs. The authors list the resulting risks. These include content exposure, re-identification from metadata, retention or logging by providers, cross-border processing, and secondary use without explicit consent. The system relies on provider defaults such as standard transport security. It adds no pseudonymization, on-device inference, or organization-managed encryption. The authors disclose these risks to users and treat privacy as a first-class constraint for future work. They also note that such controls could be critical for enterprise adoption.

What This Case Study Shows About Building Multi-Agent Systems​

Centralized supervision is a deliberate tradeoff​

The authors conclude that central supervision simplifies design but limits scalability. Routing every exchange through one coordinator made the system modular and easy to follow. The distributed extension spreads sessions across supervisor instances. Within a session, every turn still passes through one supervisor.

Usability results reflect the whole system​

The user-study results suggest that participants found conversational, supervisor-led scheduling usable and trustworthy. These results come from a 20-person study. The study reports these results for its multi-agent implementation. It does not establish how much of the result is attributable to multiple agents rather than tools, prompts, workflow design, or other system choices.

Test in the languages your users speak​

An English-only test set would have reported a perfect score and hidden the main weakness. The multilingual table shows that degradation concentrates in edit and delete operations. Teams building for non-English audiences should include those languages and those operations in the evaluation set from the start.

Put hard rules in code​

Conflict checks, ID-based targeting, and agent names are all contract details. The paper shows them expressed in prompts. Enforcing them in tools and generating routing definitions from one source would close several of the gaps documented above.

Know when the pattern fits​

A supervisor with specialists suits workloads with distinct operations that need different tools and carry different risks. It also helps when each operation needs its own evaluation, or when destructive actions need tighter controls than reads. A single agent with several tools may be simpler when operations share context and carry similar risk.

This research uses the same framework-built, tool-using agent design that WSO2 Agent Manager is built to operate and govern across frameworks such as LangChain.

Conclusion​

ScheduleMe shows that a supervisor with four specialists can handle a natural-language task with real side effects. In the user study it achieved high self-reported task success and strong usability ratings. It also shows where such systems still break. Non-Latin scripts, similar event titles, and the absence of persistent memory produced the documented failures. The privacy gaps remain open by the authors' own account.

The full paper includes the agent prompts, the error analysis, and the future-work roadmap. Read it on arXiv, or find it with related work on the WSO2 Research publications page. For the authors' shorter summary, see Srinath Perera's ScheduleMe post on the WSO2 blog.