Skip to content

Privacy Analysis of AI Chat Agents: DIY Check Guide

Skip the leak headline pile-on. Run a 15-minute DevTools HAR check modeled on the PoPETs study and see what your chat session actually sends.

7 min readBeginner

By the end of this guide you’ll run a 15-minute browser check that shows whether your ChatGPT, Claude, Grok, or Perplexity session is shipping conversation URLs, auto-titles, or share links to third-party ad/analytics domains – the same class of leak measured in Prompt like a Butterfly, Sting like a Tracker: A Privacy Analysis of Web and Mobile Conversational AI Agents (PoPETs; also tracked as LeakyLM).

Headline threads already told you “chat UIs grew trackers.” Useful. Incomplete. You still don’t know what this login, this cookie banner choice, and this build send. That gap is the whole point of Method B below.

What the paper actually proved (60-second version)

6 of 9 web clients handed conversation-derived artifacts to third parties. So did 3 of 8 Android clients. Artifacts meant URLs, titles, prompts, screenshots – often sitting next to persistent IDs.

The IMDEA Networks team (Oliveira, Sanchez, Vallina-Rodriguez, and co-authors) hit nine web products and eight Android apps: ChatGPT, Claude, Grok, DeepSeek, Perplexity, Gemini, Microsoft Copilot, Meta AI, Mistral’s Le Chat. May 2026, Spain. Chrome DevTools/CDP + HAR on web; Androguard plus instrumented AOSP on Android. They mapped 44 third-party organizations. Every service baked in at least one advertising or tracking service (ATS). Google Tag Manager showed up in relationships for six providers; GA, Meta Pixel/CAPI, and TikTok Analytics appear in their third-party tables. Full record: IMDEA handle and the paper PDF.

Grok was the loudest case in write-ups: guest chats public by default; TikTok-facing screenshot/Open Graph style content; Meta Pixel paths carrying conversation ID, title, full URL, share IDs with _fbp. Canaries on shared Grok threads got fetched from infrastructure in 14 countries – about 65.7% U.S.-origin – even when chats started in the EU. Authors notified vendors and European DPAs and read the behavior against GDPR and ePrivacy.

Read that as a snapshot of May 2026 lab conditions, not an eternal leaderboard. UIs move.

Method A vs Method B

Approach Time What you get What you miss
A – Paper + LeakyLM site 20-40 min Vendor matrix, legal framing, web+Android picture Nothing guaranteed about your tier, region, or today’s banner code
B – DIY DevTools HAR ~15 min Live requests under your account and consent choice Pure server-side forwards the browser never sees

Method A is fine for context. Method B answers the only question that changes how you type tonight: does this session leak?

It’s a stripped web pipeline in the spirit of the paper’s DevTools/HAR work. Android replication stays out of scope on day one – different lab.

DIY walkthrough: catch conversation leaks in DevTools

Fresh browser profile (or a strict container). One chat site. Ten minutes. Don’t paste real medical, legal, or HR text while testing.

1. Prep the capture

  1. Chrome/Chromium temporary profile.
  2. Open one assistant first: chatgpt.com, claude.ai, grok.com, or perplexity.ai.
  3. F12 → Network. Enable Preserve log. Start on Fetch/XHR; clear the filter when you hunt pixels.
  4. Pick the consent path before the first message: Reject non-essential or Accept all. The study’s surface changed with that choice – run both when you want the delta.

2. Trigger a boring, unique canary

Send something no one else will search: Explain the color of a 7-sided purple widget named QWE-4417. Wait for the auto title. Copy the address-bar conversation URL/id.

Weird canaries feel silly. That’s the feature. If QWE-4417 shows up on a third-party host, you aren’t squinting at coincidence.

3. Hunt third-party hits

Search the Network panel for pieces of your conversation id or path (/c/, /chat/, UUID fragments) and for the auto-title string.

Hosts that mattered a lot in the paper’s tables: google-analytics.com, googletagmanager.com, doubleclick.net, facebook.net / connect.facebook.net, analytics.tiktok.com. Open the ugly ones. Query string, body, cookies (_fbp, _ttp).

Turns out the preliminary LeakyLM disclosure pages already showed free logged-in ChatGPT shipping conversation URL + page title to Google Analytics; Claude’s Meta pixel gated on non-essential consent in that earlier matrix; Grok stacked multiple ATS paths. Use those as “what bad looks like,” then trust your HAR over memory of a blog screenshot.

Pro tip: Network → ⋮ → Export HAR, then Ctrl+F the canary offline. Closer to how traces get reviewed than scrolling a live waterfall.

4. Test share / guest paths separately

If Share or guest exists, mint one link. Open it logged-out. Full thread visible? That’s the access-control gap. Trackers that only receive a URL can still fetch content. In the study, Grok guest threads started public; circulating a link and flipping visibility later is not the same as un-ringing a bell.

5. Score your session

Red: conversation URL, title, prompt snippet, or screenshot metadata to a third-party ad/analytics host beside a stable ID cookie.
Yellow: first-party analytics only – or third-party only after Accept all.
Green for this test: no conversation artifact on third parties under Reject all. Still not a lab audit.

Web ≠ app. Paper saw that split clearly. Your laptop check still answers the daily driver question: web UI leakage under my cookie choice.

Edge cases the headlines skip

Titles are the silent doxx. You never hit Share. The model still writes a tight title and analytics loves page_title. Study figure: title leaks on 3/9 web clients toward parties including Meta, TikTok, and DoubleClick, with most of those title leaks only after non-essential cookies (about 88.9% of that title-leak set in their counts). Health and comp summaries are short enough to travel even when the full prompt does not.

Consent is not a master switch. Some ATS paths wait for Accept all – Grok’s denser stack looked heavily accept-gated in their counts. Others still fire on free logged-in browsing (ChatGPT → GA URL/title pattern on LeakyLM). One banner click, one story. You need both captures or you false-comfort yourself.

Ad blockers stop browser pixels. Not necessarily the server path. Grok’s server-side GTM / event forwarding can carry conversation URL, title, and synced _fbp/_ttp-style IDs past client blockers. HAR clean + Share enabled? Treat the URL itself as the payload.

Paid consumer tiers weren’t the paper’s hero variable the way consent and share defaults were. If a vendor quietly strips ATS on Pro tomorrow, great – re-run the capture. Don’t skip Method B because the badge says Plus.

Ever pasted a share link into Slack “just for the team,” then rotated the product’s visibility toggle and felt safe? Who else already fetched it – CDN logs, unfurl bots, a teammate’s malware scanner – is the part dashboards won’t show you.

Four fixes you can apply today

  • Reject non-essential cookies on each assistant, then spot-check with the steps above. Re-check after big UI ships – banner code drifts; May 2026 is a pin, not a promise.
  • Temporary / incognito chat modes for health, money, HR, secrets. They cut history/training in many products. They do not auto-strip third-party tags. First session still deserves a Network glance.
  • Never treat Share as private. Redact, prefer screenshots you control, revoke in the share manager after.
  • Web + hardened browser when you care: fewer opaque mobile SDKs, easier HAR. Work secrets → enterprise/API with a real DPA and no-train language. Consumer free/pro HTML shells were the study’s focus.

Next tabs if you keep going: product temporary-chat toggles, Claude data controls, local-first clients when prompts shouldn’t sit inside third-party HTML at all.

Open one assistant, reject non-essential cookies, send the purple-widget canary, export the HAR. Ten minutes. Marketing pages won’t match that file.

FAQ

Does rejecting cookies stop the leaks from A Privacy Analysis of Web and Mobile Conversational AI Agents?

No. It shrinks the surface. Some ATS paths only wake after Accept all. Others still attach conversation URL/title under stricter choices. Product-dependent. Measure both.

I only use the mobile app – does the DIY browser test still matter?

Partly. Web and Android diverged in the study (artifact leaks on 3/8 Android clients; different SDK sets). A clean desktop HAR does not clear the app.

It still teaches domain names and parameter shapes you’ll recognize later on a phone proxy. Phone-only for now? Avoid guest/public share defaults and kill ad personalization IDs at the OS level while you line up a proper capture.

Is Grok the only risky one?

No. Every evaluated service had at least one ATS. Grok mixed dense trackers, public-by-default guest permalinks, and TikTok-facing screenshot/OG style content, so it ate the headlines. ChatGPT, Claude, and Perplexity still showed conversation URL or identity-adjacent flows in the paper and the earlier LeakyLM write-up (Perplexity’s Meta Pixel noted as discontinued around Apr 2026 on that site – verify live). Premium branding is not a Network-tab argument.