Multi-Task RL Training: Consolidating Specialized Agents into One Model
This is a guest post originally published on OpenPipe.
At OpenPipe, we’ve spent the last year pushing the boundaries of what’s possible with reinforcement learning for language models. We’ve trained models that beat o3 at email retrieval with ART-E. We’ve trained summarizers that outperform GPT-4 with Summary-RL. Each success has been a specialized model, trained for a single task.
But managing these models can quickly complicate the infrastructure picture. Each model needs its own infrastructure, its own deployment pipeline, monitoring etc. If a customer wants both a search agent and document summarization, do they need to run two separate models? What about when they need five different capabilities? Ten? A hundred?
This led us to a fundamental question: Can we train a single model to excel at multiple, completely different tasks without performance penalties?
Experiment Setup: Diverse Tasks
To properly test multi-task training, we needed tasks that would truly stress the system. We deliberately chose two tasks that couldn’t be more different:
ART-E: An email search agent that requires:
- Multi-turn conversations with complex state management
- Tool use with error handling
- Optimization for search accuracy
- Context window management across long conversations
Summary-RL: A document summarizer that requires:
- Single-turn processing
- Pure text generation without tools
- Optimization for conciseness and accuracy
- No conversation state or memory
These tasks are architectural opposites. ART-E is stateful, tool-heavy, and conversational. Summary-RL is stateless, tool-free, and single-shot. If one model could excel at both without significant degradation, it would strongly validate the approach for other combinations of tasks.
The Experiments
We trained Qwen-2.5 14B using three different approaches, each for one epoch:
- Single-task baseline: Traditional training on each task individually
- Sequential multi-task: Train on first task fully, then train on the other task
- Interleaved multi-task: Alternate between tasks during training - one step of ART-E, one step of Summary-RL, repeat
The single-task baselines establish the target performance for each task when trained in isolation. The sequential and interleaved methods then test whether we can achieve similar performance when training on both tasks together.
Results
| Data | ART-E Accuracy | Summary-RL Accuracy |
|---|---|---|
| ART-E only | 91% | - |
| Summary-RL only | - | 76% |
| Sequential (ART-E → Summary-RL) | 88% | 76% |
| Sequential (Summary-RL → ART-E) | 90% | 75% |
| Interleaved | 94% | 75% |
Catastrophic forgetting is nearly non-existent. When we trained sequentially, the first task only dropped by 1-2% after training on the second task. This is remarkable - we expected much larger degradation, especially given how different these tasks are.
Task order barely matters. Our bidirectional sequential experiments show that whether we trained ART-E first or Summary-RL first, the results were nearly identical.
Interleaved training demonstrates superior performance. Our interleaved approach achieved the best ART-E performance at 94% while maintaining competitive Summary-RL performance at 75%. While this could be within statistical noise, it certainly demonstrates that multi-task training doesn’t require sacrificing performance.
Practical Implications for Production Systems
Multi-task training offers immediate practical benefits.
Infrastructure Simplification. Instead of managing multiple models, deployments, and pipelines, you maintain one. This dramatically reduces operational complexity and cost.
Incremental Capability Addition. The minimal forgetting we observed (1-2%) suggests you can add new capabilities to existing models without full retraining. Need to add summarization to your email search agent? Just continue training on the new task.
Resource Efficiency. A single model handling multiple tasks is more efficient, especially when request volume doesn’t saturate the model’s throughput. Previously, self-hosted models meant paying for idle GPU time between requests. Now, with multiple capabilities in one model, you’re more likely to utilize that compute fully. This makes self-deployment economics more similar to pay-per-use APIs.
Conclusion
Our early experiments demonstrate that multi-task RL training offers a practical path to consolidating specialized models without the expected performance penalties, potentially simplifying how we deploy AI agents in production.
If you’re interested to build your own product using ART, join us on Discord. And if you’re an enterprise interested in working with our team to train your own proprietary agents, let’s chat and see how we can help!