Agent File Understanding (Vision & Document Text)
Before this capability, an agent could see that a customer had attached a file, but not what was in it — it had to ask the customer to describe a screenshot or type out what a document said. File understanding closes that gap: when it’s on for a tenant and channel, the agent can look at a customer’s photo directly and read the text pulled out of an attached document, and answer using that content in the same turn.
There are two independent tenant flags:
agent_file_understandingenables file understanding for web chat.agent_file_understanding_emailenables file understanding for email.
Both flags ship off. They’re part of DEFAULT_DISABLED_FLAGS, so a tenant with no override row reads as disabled rather than silently on — nothing changes for a tenant until someone turns the relevant flag on for them. Turning on web chat file understanding does not turn on email attachment processing, and does not enable the extra vision or document spend that can come from reading email attachments.
What the flag enables
With the relevant flag on for a tenant:
- Photo / image — A vision part for the image. For jpeg, png, and supported webp images, the platform first tries to send a stored 512px copy inline, or create that 512px copy on demand when it is missing. The 3 MB limit applies to whether the resulting image bytes can travel inline. If a copy cannot be made, the original image bytes are tried inline instead, under the same limit. If the bytes cannot be inlined, it signs a link — for safe internal images only. If that cannot be done either, the image is dropped.
- Document — The extracted text pulled from the file, not a summary of it.
agent_file_understanding applies to web chat. agent_file_understanding_email applies to email. They are separate switches: enabling one channel does not enable the other.
On both channels the platform searches the last 10 inbound messages, not only the one being answered. Web chat depends on that: the widget sends the file on its own message and the customer types the question on the next one, so the answered message rarely holds the file. Email works the same way.
With the relevant flag off, nothing changes for that channel: the agent’s instructions still tell it that it cannot see images, and that statement stays true until the matching flag is turned on for that tenant.
Images: what’s supported
An image is sent as a vision part. The platform resolves that vision part in this order:
- For jpeg, png, and supported webp images, send a 512px copy inline: use the stored 512px copy when one exists, or create that 512px copy on demand when it is missing.
- If those resulting bytes can travel inline within the 3 MB limit, send them inline.
- If a 512px copy cannot be made, try the original image bytes inline under the same limit; if the bytes still cannot be inlined, use a signed link — only when the image is a safe internal file.
- Otherwise, send nothing for that image.
When an email image of about 50 KB or more ends up in step 4 and so cannot be shown to the AI, the platform now raises an internal alert, so the gap is visible to the team rather than silent. Logo-sized email images that are dropped on purpose do not raise this alert.
Supported formats are jpeg, png, and webp. TIFF and AVIF files are converted to a JPEG copy, and that copy is sent. Other formats are handled differently depending on the channel:
- Web chat: a GIF, HEIC, HEIF, or BMP produces a note asking the customer to resend the picture as JPG, PNG, or WebP.
- Email: a GIF stays silent — no note is produced and nothing is sent. A HEIC, HEIF, or BMP gets the same resend note as on web chat, unless it is logo-sized (under about 15 KB); like any logo-sized email image, it is then dropped silently.
None of these are forwarded and rejected downstream. Migrated external images that cannot be inlined are also dropped rather than sent as unreadable raw URLs.
A file that can’t be used safely is never sent to the model. This covers a damaged file, a file that decodes to a picture too large for the model (longest side about 5,536 px), and a file whose copy fails and is too large to send whole. In these cases the AI asks the customer once to send the file again, and tells them the supported formats and the 25 MB limit. If a second file also can’t be opened, the AI stops asking.
Model capability works the other way round from what you might expect. An image is withheld only when the resolved model is explicitly tagged as having no vision — and only one provider reports that capability at all, so absent capability data means send the image. If the provider then refuses it, the turn is retried once without the image rather than failing. A model reached through an explicit pin is treated exactly like any other: the model’s own record decides, and no plan or tier is consulted.
On an explicitly non-vision model, every image part is replaced in place by a short placeholder note, whether or not the message also carried text, so the agent knows to say it is answering without having seen the picture. On a provider refusal, the image is replaced instead by a note asking the customer to send it again as JPG, PNG or WebP, under 25 MB. An image dropped earlier as an email GIF, a logo-sized email image, or an unresolvable file produces no part at all, and therefore no placeholder note (the resend request described above is separate from this note).
In the inbox, a file the AI couldn’t analyze can show one of these tooltips:
- “Couldn’t read this file — it may be damaged”
- “Too large to analyze”
- “This picture format can’t be analyzed — open the file to view it”
For email, an inline image that the body displays with a cid:... reference is treated as a customer file when it is 15 KB or more. This covers screenshots pasted into the email body from iPhone Mail or Outlook, which arrive as cid: images rather than ordinary attachments, so the agent can read them when agent_file_understanding_email is on. The exception is an image that sits in the signature: signature logos and banners stay out, as do inline images under about 15 KB, which are treated as displayed email content rather than something for the agent to read. Real customer attachments that happen to have a Content-ID, including Gmail or Apple Mail attachments with unreferenced Content-IDs, are also kept and can be read.
Documents: extracted text, not a summary
A document is sent as the text already extracted from it during attachment processing — the same text that’s clamped to 40,000 characters and stored on the attachment once extraction succeeds. It is not the short AI-generated summary that’s shown elsewhere for that attachment.
This distinction is deliberate: a summary is a handful of bullets chosen before anyone knew what the customer would ask. Answering from the summary means the agent can only ever look as far as whatever the summarizer happened to pick, and it would still confidently answer even when the real question needed something the summary left out. The extracted text lets the agent answer a question it couldn’t have anticipated, which is why documents are read as text and not summaries.
Because the extracted text is stored going forward but nothing was backfilled, a document attached before this capability shipped has no stored text and simply sends nothing — the agent behaves as it did before the relevant flag existed for that one attachment.
Current limits
- Web chat and email are controlled by separate flags.
agent_file_understandingenables web chat only.agent_file_understanding_emailenables email only. Turning on web chat does not enable email attachment processing or the vision/document spend associated with email attachments. - Both flags are off by default. A tenant needs an explicit override for each channel where file understanding should be enabled.
- The window is the last 10 inbound messages, on both channels. A file the customer sent a few messages ago is still reachable; one older than that window is not. The agent is explicitly told never to refuse a file just because it arrived earlier in the conversation.
- Documents are read as extracted text, never a summary. If extraction hasn’t produced text for a document, the agent gets nothing for it, even if a summary exists.
- This does not apply to voice or any other channel beyond the enabled web chat or email path.
- A specialist does receive the file, passthrough included. The same parts composed onto the orchestrator’s turn are handed to the specialist on every invocation — deliberately, because on the passthrough path the specialist’s reply is what the customer reads, and a file that reached only the orchestrator would be invisible on the reply that ships.
- Files the model can’t read are never sent to it. Damaged files, pictures too large for the model, and unsupported formats are kept off the turn. On web chat the AI asks the customer to resend unsupported formats (GIF, HEIC, HEIF, BMP); on email those cases can stay silent. For damaged or oversized files the AI asks once for a new copy, and a second unopenable file ends the asking. The agent is also told whenever at least one usable file part made it onto the turn that it may be missing something, so it won’t claim to have seen a file it didn’t get.
- Email inline
cid:...images are filtered by size and position. An inline image the body displays with acid:...reference counts as a customer file when it is 15 KB or more, including screenshots pasted from iPhone Mail or Outlook, unless it sits in the signature. Signature logos, banners, and inline images under about 15 KB are left out. Attachments with a Content-ID that the body does not reference are still read when the email flag is on.
Verifying or requesting the flag for a tenant
File understanding is enabled per tenant through the existing tenant flag-override mechanism (the same mechanism used for other tenant-scoped flags) — there is no dedicated Settings toggle for it yet. If a customer reports the agent isn’t reading their attachment, or support needs to confirm whether a tenant has this on:
- Confirm the channel:
For web chat, checkagent_file_understanding.
For email, checkagent_file_understanding_email. - Confirm the matching channel flag is enabled. Enabling
agent_file_understandingfor web chat does not enable email attachment processing. - Confirm the attachment is on one of the last 10 inbound messages in the conversation. Older than that and it is out of the window on either channel.
- For email, if the image is an inline
cid:...image (for example a screenshot pasted from iPhone Mail or Outlook), confirm it is 15 KB or more and is not in the signature. Signature logos, banners, and inline images under about 15 KB are intentionally left out. If a non-GIF image of about 50 KB or more could not be shown to the AI, check for the internal alert raised for it. - If it’s a document, confirm the attachment finished extraction (
text_extracted) rather than failing to extract — a document with no extracted text sends nothing regardless of the flag. - To check or change whether either flag is on for a tenant, escalate to engineering/support ops. The flags are visible and overridable on the superadmin Feature Flags page; there is no tenant-admin setting that reflects them.
What the agent is told
When a turn carries at least one usable file part, an additional block is appended to the agent’s prompt — only for that turn. It tells the model:
- An image may have been replaced by a short note saying it could not be shown. This note applies to models that cannot see, so the agent should say it is answering without having seen the picture.
- If the note instead says the file could not be opened and asks for it again, the agent says it could not open the file and asks the customer once to send it again, using the formats and limits the note gives. If it already asked, or the note says not to ask, it says it cannot open the file and continues without it.
- A document arrives as extracted text, so the agent should not claim to have “seen” the document itself.
- A file often arrives earlier in the conversation than the words being answered, so it must never refuse a file just because it came earlier — but a file nothing could be read from, and one older than the recent-message window, are not there at all.
With the relevant flag off, none of this block is added for that channel, and the platform-level instruction that the agent cannot see files stays exactly as it was.