There is a very specific moment when a text-only chatbot becomes annoying.

You are trying to explain what’s wrong with something on your screen, so you type:

There is a red button near the bottom-right, and above it there’s a box with some numbers, and the second number looks wrong…

At some point you realize you could have saved both yourself and the AI a lot of trouble by attaching the screenshot.

That’s the basic appeal of multimodal chat.

Instead of forcing every piece of information through text, users can say what they mean and show what they’re talking about in the same conversation.

Upload a dashboard and ask why a metric looks strange.

Send a product photo and ask which model it is.

Drop in a chart and ask what changed.

Show an error screenshot instead of manually transcribing it.

Upload a receipt and ask where most of the money went.

For developers, this changes the chat interface from a text-generation feature into something much closer to a general-purpose analysis layer.

LLMAPI currently exposes multiple vision-capable models alongside its regular language models, giving developers a way to build conversations where text and images can become part of the same request. The current model catalog includes vision-capable options from several model families, with capabilities ranging from ordinary image description to OCR, document understanding, charts, screenshots, and multimodal reasoning. Browse the current vision-capable models on LLMAPI

But adding an Upload button is the easy part.

The interesting question is what happens after somebody clicks it.

A picture changes the question before the model answers it

Let’s build an imaginary app.

We’ll call it Fixly, an AI assistant that helps people troubleshoot home electronics.

Without image input, somebody might write:

My router has a weird light on it. What does it mean?

The assistant now needs several pieces of information.

Which router?

Which light?

What color?

Is it blinking?

Where is it located?

Does the device have labels?

The conversation becomes an interrogation.

Now give the user an image upload.

They send a photo and ask:

What’s this blinking orange light?

The model can inspect the image while interpreting the question.

The user hasn’t become better at describing networking hardware.

They don’t have to.

That reduction in translation effort is one of the biggest practical advantages of multimodal interfaces.

Visual-language research has been moving toward exactly this kind of interaction for several years. DeepMind’s Flamingo research was particularly influential because the model was designed to accept sequences where images and text are interleaved instead of treating vision as an isolated image-classification job. The researchers tested it across visual question answering, captioning, and other tasks where understanding the relationship between the prompt and the visual content matters.

Later systems pushed that idea further toward the assistant-like experience we now recognize as multimodal chat.

The LLaVA research paper on visual instruction tuning, for example, connected a vision encoder with a language model and specifically trained the resulting system to follow natural-language instructions about images.

That is an important distinction.

The model doesn’t merely produce:

Router. Black. Electronic device.

It can respond to:

Which cable should I check first?

or:

Is anything obviously connected to the wrong port?

The image provides evidence.

Language tells the model what to do with that evidence.

“Analyze this image” is actually dozens of different jobs

We tend to use image understanding as if it describes one ability.

It doesn’t.

Consider these requests:

What’s in this photo?

Read the serial number.

Which shirt is darker?

Why is this chart dropping?

Find the typo in this screenshot.

Turn this handwritten checklist into JSON.

What part of this circuit diagram connects to the battery?

Compare these two product images.

They all contain an image.

Almost everything else about them is different.

A multimodal model may have to combine:

• object recognition;
• OCR;
• spatial understanding;
• chart interpretation;
• visual comparison;
• language reasoning;
• domain knowledge;
• structured extraction.

This is one reason model selection matters.

LLMAPI’s current Qwen3-VL 8B Instruct listing, for instance, describes support for text and image input alongside visual question answering, OCR-style extraction, screenshot analysis, and general multimodal chat. See Qwen3-VL 8B Instruct on LLMAPI

GLM 4.6V goes further into complex documents, diagrams, charts, scanned pages, and multimodal reasoning. Its LLMAPI listing describes image, text, document-layout, and visual-OCR capabilities within a conversational model. See GLM 4.6V on LLMAPI

So before choosing a model, we’d ask a more useful question than:

Does it support images?

Ask:

What kind of thinking do we expect it to do with those images?

Let users point instead of describe

One of the nicest things multimodal chat does is remove vocabulary requirements.

Imagine building support software for a complicated analytics dashboard.

A user doesn’t know that the object they’re looking at is called a cohort-retention heatmap.

They upload a screenshot and write:

Why does this part suddenly get darker?

That is enough.

Or someone working with machinery sends:

This thing is leaking. Is this the valve you meant?

A fashion app gets:

Find me something in this kind of green.

A gardening assistant gets:

What are these white spots?

An ecommerce support bot gets:

This is what arrived. Is it the same product I ordered?

Humans communicate like this constantly.

We point.

We show.

We circle.

We say this one, that bit, the thing on the left.

Text-only interfaces quietly ask users to convert visual information into prose before the software can help them.

Multimodal chat removes part of that conversion layer.

And that makes the interface useful even when the underlying model hasn’t become dramatically better at language.

An upload isn’t useful until you preserve the question around it

Suppose a user uploads this image:

dashboard-august.png

You could immediately ask the model:

Describe this image.

You would probably get something.

You would probably also miss the reason the user uploaded it.

The better request is:

Our conversion rate normally sits around 4%. Look at this dashboard and tell me what changed this week.

Now the model has:

Visual evidence

The chart.

Background context

Conversion normally sits around 4%.

Instruction

Explain what changed this week.

That’s multimodal prompting in its simplest useful form.

The quality of image analysis often depends heavily on the accompanying language because the prompt directs attention.

“Analyze this chart” is broad.

“Compare July and August conversion rate and identify the largest week-over-week decline” gives the model a job.

We see this relationship clearly in research benchmarks too. ChartQA was created specifically because answering questions about charts requires more than recognizing what appears visually. Many questions require logical or arithmetic reasoning over the visual data.

So if your product has domain context that can make the question more precise, send it.

Don’t make the model guess why the image matters.

What does the API flow look like?

At application level, a multimodal conversation is still fairly understandable.

A user might send:

Can you tell me why this graph looks strange?

plus:

analytics-dashboard.png

Your frontend first handles the upload.

Then your backend prepares a request containing both the text instruction and the image input in the format supported by the selected model/API route.

LLMAPI’s regular conversational APIs include an OpenAI-compatible /v1/chat/completions endpoint as well as a /v1/responses endpoint for Responses-compatible clients. See the LLMAPI Chat Completions documentation

LLMAPI also exposes Gemini-native multimodal routes for Gemini-group keys, including requests that combine text with image input. See LLMAPI’s Gemini multimodal API documentation

The exact payload depends on the protocol and model you choose, but conceptually you’re sending something like:

{
  "role": "user",
  "content": [
    {
      "type": "text",
      "text": "Why did conversion drop here?"
    },
    {
      "type": "image",
      "image": "uploaded-dashboard.png"
    }
  ]
}

Then the model responds in text:

The largest decline appears between August 12 and August 15. Conversion falls from roughly 4.1% to around 2.8% while traffic remains relatively stable, so the problem may be closer to checkout behavior than acquisition volume.

Your app can render that exactly like another chat message.

From the user’s perspective:

1. Ask.
2. Upload.
3. Get an answer.

Most of the architecture sits quietly underneath.

URLs, uploaded files, and Base64 all solve the same basic problem

Your model somehow needs access to the image.

Applications generally handle that in one of a few ways.

Send an accessible image URL

Your app uploads the image to storage and sends the model a temporary or accessible URL.

This works nicely when:

• files are already stored remotely;
• you want to reuse the same image across requests;
• images are large enough that stuffing them into JSON is awkward.

Encode the image

Another option is converting image bytes into a Base64/Data URL representation where the target API supports it.

This can simplify small prototypes because you don’t necessarily need permanent external storage first.

It also makes requests heavier.

Keep your own upload layer

For production applications, we generally prefer thinking of uploading and model inference as separate systems.

Your upload service can handle:

• file-size limits;
• supported formats;
• virus scanning;
• access controls;
• temporary storage;
• image normalization;
• retention;
• deletion.

Then the AI request receives a clean, approved image reference.

That separation becomes increasingly useful once users start uploading real business documents instead of cute pictures of dogs.

One image can support an entire conversation

The first response is rarely the interesting part.

Imagine the user uploads a dashboard and asks:

What’s unusual?

The assistant identifies a traffic spike.

Then the user says:

Ignore traffic. What about revenue?

Then:

Compare the second and fourth weeks.

Then:

Could this be caused by the lower average order value?

These are follow-up questions about the same visual context.

That’s where multimodal chat becomes meaningfully different from a one-shot image-analysis endpoint.

Your conversation layer needs to retain enough context that later questions make sense.

Depending on the API protocol and architecture, that can mean preserving and resending the relevant message history, maintaining references to previously uploaded files, or storing your own structured conversation state.

We wouldn’t assume that “the model remembers the image forever.”

Your application should know which uploaded visual belongs to which conversation.

A useful internal record might contain:

conversation_id
message_id
image_id
storage_url
upload_time
mime_type
model_used

Then if the user asks:

What about the previous screenshot?

you can resolve previous screenshot into an actual asset.

That tiny piece of state management prevents a lot of very confusing AI conversations.

Multiple images make comparison much more useful

Single-image analysis gets most of the demos.

Real applications often need comparison.

Upload:

before.jpg

and:

after.jpg

Then ask:

What changed?

That works for more domains than you might expect.

Ecommerce

Are these two products actually the same color?

Design review

Which version gives the headline more visual emphasis?

Property inspection

What damage appeared between these two inspections?

Manufacturing

Compare the defective component with the reference component.

Analytics

Which dashboard shows better retention?

Education

Compare my solution with the worked example.

QA testing

What changed between the old and new interface?

The challenge is making references explicit.

Instead of:

What’s different?

we’d prefer:

Image 1 is the current checkout page. Image 2 is the redesigned checkout page. Compare layout, visual hierarchy, form complexity, and CTA prominence.

That prompt gives the model both labels and criteria.

The less ambiguity you leave around image roles, the more useful the comparison becomes.

Screenshots are secretly one of the best multimodal inputs

Photos get all the attention because computer vision sounds very futuristic.

Screenshots may be more useful for everyday software.

Think about how often users need help with:

• error dialogs;
• spreadsheets;
• dashboards;
• settings pages;
• code editors;
• website layouts;
• forms;
• invoices;
• admin panels.

With text-only support, someone writes:

It says something about authentication and then there is a code 401 and underneath something about a token.

With image input:

Why am I getting this?

Upload.

Done.

A vision-capable model can combine visible text, interface structure, and the user’s question.

LLMAPI’s current vision model catalog includes models specifically described as suitable for screenshots and interface understanding. Qwen3-VL 8B Instruct, for example, is positioned for general multimodal chat, screenshots, OCR, and visual reasoning, while GLM 4.6V supports more complex layouts and document-oriented inputs.

That opens up a particularly nice support workflow.

The model can:

1. Read the error.
2. Identify the application state.
3. Explain the likely cause.
4. Ask for missing information.
5. Suggest the next troubleshooting step.

One screenshot can replace three rounds of “what exactly do you see?”

OCR inside multimodal chat is useful — within limits

Vision models can often read text inside images.

That means a user can upload:

• receipt;
• menu;
• sign;
• label;
• screenshot;
• handwritten note;
• presentation slide.

Then ask questions about the content.

For example:

How much tax did I pay?

Translate the second paragraph.

Turn these notes into tasks.

Which item on this menu is vegetarian?

What’s the error code?

This overlaps with OCR, but we’d still distinguish general visual chat from specialized document extraction.

If you’re processing 100,000 standardized invoices and need reliable fields such as:

supplier_name
invoice_number
subtotal
tax
total
due_date

a document-specific OCR pipeline is usually a better engineering choice.

If somebody casually uploads one invoice and asks:

When is this due and what’s the total?

multimodal chat may be perfectly reasonable.

The user experience drives the architecture.

Interactive question answering and high-volume structured extraction are related problems, but they aren’t identical.

Charts expose the difference between seeing and reasoning

Here’s a good test for a multimodal assistant.

Upload a chart and ask:

What is the highest bar?

That’s mostly perception.

Now ask:

If Q3 continued growing at the same rate as Q2, what would you expect Q4 to be?

We’ve added arithmetic and inference.

Or:

Traffic increased, but revenue didn’t. What metric here might explain that?

Now we need relationships between multiple values.

This is why vision-language evaluation has expanded far beyond image captioning.

The MMMU benchmark contains 11,500 multimodal questions spanning 30 subjects and visual formats such as charts, diagrams, maps, tables, musical notation, and scientific figures. Its purpose is to test whether models can combine visual perception with domain knowledge and deliberate reasoning.

That combination is closer to what actual applications need.

A business user doesn’t upload a dashboard because they want the model to say:

I see a blue line.

They want:

Why is the blue line doing that, and should I care?

Documents are visual too

We often mentally separate image analysis from document analysis.

Models don’t get that luxury.

A photographed report can contain:

• paragraphs;
• tables;
• charts;
• footnotes;
• logos;
• callout boxes;
• page structure.

The meaning may depend on where those elements sit relative to one another.

Researchers working on multimodal document question answering have found that text-only retrieval can miss important visual evidence. The 2025 MMDocRAG benchmark was designed around exactly this problem: answering questions that require evidence across multiple document pages and multiple modalities, including text and visuals.

This becomes relevant if you’re building:

• report assistants;
• insurance tools;
• financial analysis software;
• academic research tools;
• contract-review interfaces;
• internal knowledge systems.

A user may upload a report and ask:

Does the chart on page 6 support the claim made in the executive summary?

That requires more than extracting every word and dumping it into a vector database.

The model needs the relationship between text and visual evidence.

Make the user tell you what they care about

Open-ended prompts are seductive:

Analyze this image.

You can definitely support them.

We just wouldn’t build the whole product around them.

Specific questions usually produce more useful answers.

Compare:

Analyze this store shelf.

with:

Count the empty shelf positions and identify which product categories appear low in stock.

Or:

Look at this interface.

versus:

Check this mobile checkout screenshot for anything that could make the primary payment button hard to notice.

Or:

Analyze this document.

versus:

Find the effective date, renewal clause, and any mention of cancellation notice.

The image stays the same.

The task gets dramatically clearer.

In products we’ve worked around over the years, this is a recurring API design lesson: better input structure often improves results more reliably than clever prompt decoration.

If your application already knows the task category, use that knowledge.

A “Check receipt” feature can automatically attach instructions relevant to receipts.

A “Review UI” feature can ask about layout and usability.

A “Read chart” tool can emphasize values, axes, legends, and comparisons.

The user still talks naturally.

Your backend quietly supplies the domain frame.

You can turn visual answers into structured outputs

A multimodal chat response doesn’t have to be prose.

Suppose somebody uploads a product photo.

Your system wants:

{
  "category": "running_shoe",
  "primary_color": "black",
  "secondary_color": "white",
  "brand_visible": true,
  "condition": "used"
}

Or upload a UI screenshot:

{
  "cta_visible": true,
  "cta_text": "Continue",
  "error_present": true,
  "error_text": "Card could not be verified"
}

Or a warehouse photo:

{
  "pallets": 12,
  "damaged_boxes": 3,
  "blocked_exit": false
}

Structured outputs make multimodal models useful inside workflows rather than only inside chat bubbles.

The model sees something.

Your software gets data.

Then regular business logic can decide what happens next.

You might:

• create a support ticket;
• tag a product;
• update inventory;
• flag a compliance issue;
• populate a form;
• search another database;
• call another API.

At that point, the image becomes another input type for your application.

Give vision models tools when the image isn’t enough

Suppose a customer uploads a picture of a product and asks:

Can I still return this?

The image may help identify the product.

It cannot tell you:

• when the customer bought it;
• which return policy applies;
• whether their order is inside the return window.

This is where multimodal chat and tool use fit nicely together.

The model can inspect the image and then your application can provide tools such as:

find_product()
get_order()
check_return_policy()
create_return_request()

The workflow becomes:

1. User uploads the product photo.
2. Model identifies relevant visible information.
3. Application searches the customer’s order.
4. Return rules are retrieved.
5. Model explains the result.
6. User confirms the action.

LLMAPI’s /v1/responses interface supports tool definitions when the selected model and API mode provide that capability. See LLMAPI’s Responses API documentation

This is where “chat with images” starts becoming a real application workflow.

Vision gives the agent context from the physical or visual world.

Tools connect that context to your systems.

Don’t let a confident answer trick you into believing the pixels

Vision-language models can be very convincing when they’re wrong.

This deserves its own section because multimodal hallucinations are especially sneaky.

A text model can invent a fact.

A vision-language model can invent something and make you feel as though it saw it.

For example:

There is a warning label in the upper-left corner.

Except there isn’t.

Or:

The chart shows a 12% increase.

The chart actually shows 8%.

Or:

The expiration date reads September 18.

The tiny blurry text was unreadable.

Researchers have repeatedly found this problem in multimodal systems. HallusionBench was specifically created to test failures caused by both visual illusion and language hallucination. Its authors found that even strong vision-language systems struggled with questions requiring careful interpretation of what was genuinely present in an image.

So we’d design the interface around uncertainty.

Tell the model:

• don’t guess unreadable text;
• say when details are unclear;
• distinguish observation from inference;
• quote visible values when practical;
• ask for a clearer image when needed.

And design your workflow accordingly.

For a casual question like:

What breed might this dog be?

some uncertainty is harmless.

For:

Read the medication dosage from this label.

your tolerance should be much lower.

Zoom, resolution, and cropping matter more than people expect

Sometimes the AI isn’t failing at reasoning.

It simply can’t see the thing you’re asking about clearly enough.

Imagine uploading a 4K dashboard screenshot containing:

• 12 charts;
• 40 labels;
• a sidebar;
• filters;
• a tiny table;
• browser chrome.

Then asking:

What’s the number in row seven?

You may get much better results by cropping to the table.

This is useful UX territory.

Your application can let users:

• crop;
• zoom;
• draw a box;
• select a region;
• annotate the image.

Then send both the original question and the focused image.

Instead of:

What does this say?

you effectively give the model:

What does this region say?

Visual grounding gets easier.

It also saves unnecessary image processing when most of the screenshot doesn’t matter.

A good upload flow needs boring engineering

The AI part gets the demo.

The upload system gets production.

Before an image ever reaches the model, we’d think through:

File types

Do you accept:

• JPEG?
• PNG?
• WEBP?
• PDF?
• HEIC?
• GIF?

Model support varies, so normalize formats when necessary.

File size

Huge images:

• upload slowly;
• consume more resources;
• may hit provider limits;
• may contain far more detail than the task needs.

Resize intelligently rather than blindly destroying resolution.

Orientation

Phone images may carry EXIF rotation metadata.

Make sure the model gets the image the same way the user sees it.

Storage

Decide whether images are:

• temporary;
• attached permanently to a conversation;
• deleted after inference;
• stored in customer-controlled storage.

Access

Private uploads should stay private.

Avoid turning confidential screenshots into casually public URLs just because the model needs to retrieve them.

Retention

If somebody deletes a conversation, what happens to its uploaded images?

Your answer should come from actual storage policy rather than vibes.

Visual conversations can contain more sensitive data than users realize

People upload screenshots very casually.

A screenshot can contain:

• email addresses;
• customer names;
• account balances;
• private messages;
• API keys;
• browser tabs;
• internal URLs;
• addresses;
• health information;
• employee data.

Users may be focused on the tiny error dialog in the middle and completely miss everything surrounding it.

You can reduce risk with:

• automatic image expiration;
• metadata stripping;
• access-controlled storage;
• sensitive-data detection;
• optional redaction;
• clear upload notices;
• strict logging policies.

For enterprise applications, we’d also consider whether the image needs to be stored at all after inference.

Sometimes the best retention period is:

long enough to answer the question.

Model routing becomes more valuable once images enter the chat

A text-only application can often survive with one default language model.

Multimodal workloads vary more dramatically.

Consider three requests.

Request A

What’s in this photo?

Simple visual understanding.

A smaller, cheaper vision model may be plenty.

Request B

Read these six values from this screenshot and return JSON.

Now OCR reliability matters.

Request C

Compare this engineering diagram with the requirements below and explain any inconsistencies.

Now you’re asking for visual perception, long-context understanding, technical knowledge, and multi-step reasoning.

Using the same expensive model for all three requests may waste money.

Using the cheapest model for all three may ruin the hard one.

LLMAPI’s current catalog gives developers several vision-capable options with different model sizes, prices, context windows, and reasoning capabilities. Explore LLMAPI’s model catalog

That makes routing a useful architecture pattern:

simple photo question
→ lightweight vision model

OCR-heavy screenshot
→ model strong at visual text

complex chart/document reasoning
→ stronger multimodal model

You can route based on:

• input type;
• number of images;
• task category;
• expected reasoning depth;
• latency requirements;
• cost limits.

The user still sees one assistant.

Your backend doesn’t have to use one brain for every problem.

Test the things your users will actually upload

Vision benchmarks are useful.

Your own screenshots are better.

If you’re building ecommerce software, your evaluation set should contain ecommerce photos.

If you’re building accounting software, test financial documents.

If you’re making developer tools, use:

• terminals;
• IDE screenshots;
• stack traces;
• dashboards;
• documentation diagrams.

Create a test set from realistic inputs and write down the correct answer for each.

Then measure things such as:

Metric	Example question
Object recognition	Did it identify the correct item?
OCR accuracy	Did it read the visible text correctly?
Numeric accuracy	Did it extract the right values?
Spatial accuracy	Did it understand left/right/above/below?
Instruction following	Did it answer the requested question?
Hallucination rate	Did it invent visible details?
Structured output validity	Did the JSON parse?
Latency	How quickly did the answer appear?
Cost	How expensive was the request?

And test ugly inputs.

Blurred photos.

Dark photos.

Tiny text.

Rotated receipts.

Screenshots containing multiple windows.

Charts with bad legends.

Partially covered objects.

Real users will eventually upload all of them, plus several things you didn’t know a camera could produce.

The best multimodal answer may still be “I can’t tell”

This sounds unimpressive until you build something users depend on.

Suppose the user uploads a blurry product label.

A weak system responds:

The serial number is XQ190428.

A better system says:

I can see the serial-number label, but the characters are too blurry to read reliably. Upload a closer photo of the lower-right label.

The second answer contains less information.

It is considerably more useful.

Good multimodal UX gives the assistant permission to request better evidence.

You can even make those requests specific:

Move closer to the label.

Include the entire chart legend.

Take the photo straight on.

Upload the original screenshot instead of a photo of your monitor.

Crop around the error message.

Now the model isn’t simply failing.

It is guiding the user toward an input it can actually work with.

Start with one upload button and one very clear job

If you already have a text chatbot, multimodal support can look like a huge expansion.

It doesn’t need to start that way.

Pick one moment where users are currently forced to describe something visual.

For a support product:

Upload a screenshot of the problem.

For an analytics product:

Ask questions about a chart.

For ecommerce:

Upload a product photo.

For education:

Upload the problem you’re stuck on.

For document software:

Upload a page and ask about it.

Then measure what changes.

Do conversations get shorter?

Do users provide better context?

Do support agents ask fewer clarification questions?

Are answers more accurate?

Do people actually use the upload feature?

Once that first workflow works, add another.

Multimodal products get messy quickly when teams start with “the model can see anything.”

They’re much easier to build when you start with:

Here’s one thing our users constantly struggle to explain in words.

Let them show it instead.

Chat gets a lot more useful when users can stop typing

The humble attachment button changes the relationship between a person and an AI assistant.

Users no longer have to describe every interface.

Or copy every error.

Or manually type a chart value.

Or know the proper name for the thing they’re pointing at.

They can ask:

What’s wrong here?

Which one should I use?

What changed?

Can you read this?

Why does this look different?

and attach the missing half of the question.

Vision-language research — from Flamingo’s interleaved image-and-text work to BLIP-2’s approach to connecting pretrained vision encoders with language models and LLaVA’s visual instruction tuning — has spent years making that interaction increasingly natural.

Through LLMAPI, developers can now choose among vision-capable models and connect those capabilities to familiar conversational API patterns rather than building a separate computer-vision product for every visual question.

The frontend gets an Upload button.

The backend gets text plus visual context.

And the user finally gets to communicate the way people normally do when words aren’t enough:

“Here. Look at this.”