The AI tools privacy problem: your tasks, notes and habits are training their models

When you type a task into Todoist AI, or ask Notion AI to summarize your meeting notes, or let Mem.ai reorganize your knowledge base, something happens that isn't clearly labeled in the feature announcement: your personal data leaves your device, travels to a third-party server, is processed by a large language model, and potentially becomes part of the training data used to improve that model. You agreed to this in a terms of service document you likely didn't read. Most users don't know it's happening.

This post examines what AI productivity tools actually do with your data, what specific ToS language enables this, and what a genuinely privacy-preserving AI alternative looks like — including the on-device AI models that are now capable enough to run in a browser without sending a single byte to an external server.

What "AI features" actually means for your data

The UX of an AI-powered productivity tool is simple: you type something, you get a smart response. What happens between those two events is considerably more complex. In virtually every cloud-first AI tool, here is the actual data flow:

1. Your prompt (or the relevant content from your notes/tasks/habits) is serialized and sent to the vendor's API endpoint over HTTPS
2. The vendor's backend processes the request — often passing it through to a third-party LLM API (OpenAI, Anthropic, or Google)
3. The response is returned to your client
4. The request and response may be logged on the vendor's servers for debugging, abuse prevention, and quality review
5. Depending on the ToS, this data may then be used to fine-tune or train future versions of their AI models

Step 5 is where the privacy problem lives — and where the language in terms of service documents becomes critically important to read carefully.

What the terms of service actually say

We reviewed the privacy policies and terms of service for several major AI productivity tools in early 2025. The patterns are consistent, even if the specific language varies.

The opt-out is real, but it requires knowing it exists, finding it in settings, and actively toggling it — in most cases before the data has already been used. Users who signed up before AI features were added often never saw a prompt to adjust their AI training preferences. Data that was used before the opt-out feature existed is generally not retroactively excluded.

Mem.ai was particularly transparent: their early product explicitly marketed the idea that your notes help the AI understand you better over time. This is genuinely useful — and genuinely means your notes are being analyzed to build a model of your thinking patterns and preferences. That model lives on their servers, not yours.

Notion AI, which is built on top of third-party LLM providers (primarily Anthropic and OpenAI at various points), has its own data processing agreement, but the data still passes through Notion's servers and is subject to their retention and usage policies, separate from OpenAI's or Anthropic's data policies. Enterprise customers can negotiate zero-retention agreements; free and Plus users operate under the standard terms.

Todoist's AI features similarly route through external providers. Their privacy documentation is more specific than most — they explicitly state that AI-processed content is sent to third-party AI providers — but the processing still happens server-side, outside your device.

"The most sensitive data you have isn't your financial records or your messages. It's the accumulated record of how you spend your time, what you find important enough to write down, and which habits you're trying to build or break. That's what your productivity app knows about you."

Why productivity data is particularly sensitive

Financial data and messages get more attention in privacy discussions, but productivity data may be uniquely revealing. A year of task history shows not just what you worked on, but what you planned and never did, what deadlines you missed, and what areas of your life you were trying to improve. Habit tracking data reveals your health behaviors, sleep patterns, and daily routines. Journal entries contain your unfiltered thoughts about your relationships, work situations, and internal state.

This is precisely the kind of longitudinal behavioral data that is most valuable for training personalized AI models — and most sensitive from a privacy perspective. It's also the kind of data that, if exposed in a breach, could be used to infer medical conditions, mental health status, employment situations, and relationship dynamics.

When this data is stored on your device only, a breach of the vendor's servers exposes nothing about you. When it lives in their cloud, every security incident they have is also a risk to your most private thoughts and behaviors.

The rise of on-device AI: what's actually possible now

The objection to local AI has always been capability: server-side models are enormously powerful, while on-device models are small and limited. This was a reasonable objection in 2022. It's significantly less compelling in 2025, and for many productivity use cases, it has stopped being true.

Several compelling on-device AI models are now available for browser-based applications:

WebLLM, developed at the Machine Learning Compilation (MLC) lab at Carnegie Mellon and the University of Washington, is a runtime that compiles LLMs to WebGPU-accelerated WASM. It enables GPT-4-class inference speeds on modern consumer hardware, directly in the browser, with the model running entirely on your device. The WebLLM project has demonstrated Llama 3 and Mistral models running at 20–40 tokens per second on an M2 MacBook Pro — comparable to what you'd expect from a cloud API with moderate latency.

For productivity tasks — summarizing a note, suggesting next actions for a task, writing a daily review prompt, tagging a journal entry — models in the 1–4B parameter range are genuinely capable. They won't write you a novel or reason through a complex legal document, but they can do the kind of light, context-aware assistance that makes productivity tools feel smart.

What "privacy-preserving AI" actually means

The phrase "privacy-preserving AI" is used loosely and often misleadingly in marketing. Let's be specific about what it can mean:

• On-device inference: The model runs on your hardware. Your data never leaves your device. This is the strongest privacy guarantee. WebLLM and Chrome's Prompt API (Gemini Nano) enable this for web apps today.
• Zero-data-retention API: Your data is sent to a server-side model but not retained after the request completes. OpenAI and Anthropic offer this for enterprise API customers. It prevents training on your data but doesn't prevent the data from being processed by a third party.
• Federated learning: Model training happens on-device using your data, but only gradient updates (not raw data) are shared with the central server. Google uses this for Gboard keyboard predictions. It's meaningfully more private than central training but still involves some information leaving the device.
• Differential privacy: Mathematical noise is added to training data or model updates to prevent individual records from being recoverable. Used in combination with federated learning by several large tech companies.

Only the first category — on-device inference — provides a genuinely strong privacy guarantee for the content of your notes and tasks. The others are better than nothing, but they all involve your data leaving your device in some form.

Our roadmap for local AI features

We're building Bun Agents' AI features around on-device inference as a strict requirement, not an afterthought. Here's what we're working toward:

Near-term (mid-2025): Integration with Chrome's built-in Gemini Nano via the Prompt API for task suggestions, note summarization, and journal prompts. This works with zero downloads for Chrome users — the model is already on their device. For other browsers, we'll offer an optional one-time download of a quantized Phi-3-mini or SmolLM2 model via WebLLM.

Medium-term (late 2025): Semantic search across your notes and tasks using on-device embedding models. Rather than keyword matching, you'll be able to search for concepts and find relevant entries across your entire database. The embeddings are computed locally and stored in your SQLite database — no external service involved.

Longer-term: Personalized AI that actually knows your patterns — your recurring tasks, your most-used note tags, your habit history — because it's reading directly from your local SQLite database. No context window limitations, no data sent to a server, no concern about what happens to your behavioral data. Just AI that understands your workflow because it has access to the data it needs, and that data stays with you.

The AI features in Bun Agents will never be the most powerful on the market — on-device models are constrained by hardware in ways that cloud models aren't. But they'll be the most private. And for personal productivity data — the record of your thoughts, your goals, and your daily life — we think that's the right tradeoff to make.