Building Multimodal
AI Agents with Java
SneakerLens — an AI sneaker authenticator built live
Java Meetup · June 2026
What we'll cover in the next hour
- 01IntroWhy sneaker authentication, why this matters
- 02Agentic · Multimodal · HybridWhat these words actually mean, when to use each, when to avoid
- 03Demo architectureThe 3-step pipeline: local guardrail → identify → authenticate
- 04Code walkthroughSwitching to VS Code — the real implementation
- 05Live demoUpload real sneakers, watch the agent work
- 06Lessons learnedThe real bugs we hit building this — more useful than a clean happy path
- 07Q&A
The problem: sneaker fakes are everywhere
- →Counterfeit sneaker resale is a multi-billion dollar problem globally — and growing fast in the Indian resale market
- →Authentication today requires human experts who look at stitching, fonts, materials — knowledge that took years to build
- →What if a multimodal LLM could do a first-pass check, instantly, from a phone photo?
- →That's SneakerLens — and building it taught me more about agentic system design than any tutorial
A familiar scenario: buying sneakers off OLX
😬 Today — manual inspection
- Zoom into seller photos, squint at the logo font
- Cross-check a forum post or "spot the fake" video
- Result: gut feeling — inconsistent, unexplainable
✅ As an agent — same photos
- Step 0: local model confirms it's actually footwear
- Step 1 → 2: identify the model, then re-ground on that model's specific tells
- Result: structured verdict + evidence — every time, the same way
Notice the shape of Step 1 → Step 2: identify first, then use that identity to know what to look for. That two-step reasoning is exactly what "agentic" means — and it's what we're about to define properly.
What actually makes something "agentic"?
It's not just "calling an LLM"
A single prompt → single response is not an agent. It's a function call.
- ✓An agent exists when one model's output changes what happens next — a different prompt, a different tool, a different path
- ✓In SneakerLens: Step 1 identifies
"Nike Air Jordan 1"→ that exact string gets woven into Step 2's system prompt, changing what Step 2 looks for - ✓That's the whole trick. No framework magic — just code-controlled branching driven by a prior LLM call's output
Three tiers — only escalate when you actually need to
Most "agent" demos online are actually Tier 1 or 2 wearing a fancier name. The real signal is whether the question changes, not just the data.
Before reaching for Tier 3, check all four
If the answer to any one of these is "no," stay at a simpler tier — a workflow or a single call will serve you better than an agent.
Multimodal: not just text in, text out
What it means here
Gemini 2.5 Flash takes images + text in the same request. No separate OCR step, no separate vision model glued on with string parsing.
We send up to 4 photos (top / side / sole / label) as Part.fromBytes() alongside the prompt text — one request, native understanding of both.
Why it matters for this use case
Authentication is inherently visual: stitching density, font kerning on logos, sole texture — things a human expert looks at, not reads.
A text-only LLM literally cannot do this task. Multimodal isn't a nice-to-have here — it's the entire premise.
Hybrid: local model + cloud LLM
- →The idea: not every check needs a $$ multimodal LLM call. A cheap, fast, local model can pre-filter before the expensive call happens
- →In SneakerLens: a local ResNet18 (ImageNet) classifier checks "is this even a shoe?" before Gemini is called at all
- →Cost asymmetry: local inference ≈ free + ~0.2s. A rejected Gemini call still would've cost tokens + 5-10s of latency
When to reach for this pattern
Whenever a cheap signal can reject obviously-bad input before your expensive step runs: image quality checks, language detection, profanity filters, "is this even on-topic" gates.
When to avoid the hybrid pattern
Skip it when…
- Traffic volume is low — the engineering cost of running a second model outweighs savings
- The "cheap" model needs constant retraining/tuning to stay accurate — now you have two ML systems to maintain
- Latency budget doesn't allow a sequential local-then-cloud hop
Use it when…
- A pretrained, off-the-shelf model already solves your filter (like ImageNet for "is this a shoe")
- You expect meaningful volume of bad/abusive input
- Cost-per-call on the expensive step is high enough to justify the guardrail
SneakerLens: the full pipeline
Step 1's output (e.g. "Air Jordan 1") is woven into Step 2's system prompt — that's the agentic part.
The Java stack, end to end
🌱 Spring Boot 3.4
Web MVC + Thymeleaf for server-rendered UI. JPA + MariaDB for users & analysis history. HttpSession for auth.
🤖 Gemini Calls
com.google.genai SDK directly (API key, not Vertex). spring-ai-model's BeanOutputConverter for structured JSON output.
🧠 Local Model
DJL (Deep Java Library) + PyTorch engine running a pretrained ResNet18 — pure Java, no Python sidecar process.
The domain models
Records as the contract Gemini's JSON has to match — @JsonPropertyDescription becomes the schema sent to the model
The dynamic prompt
This is where the "agent" actually happens — Step 1's identity gets formatted directly into Step 2's system prompt string
The two-step chain
Guardrail → Step 1 → Step 2. Plus the retry-with-backoff wrapper for Gemini's transient 503s
The local guardrail
DJL Criteria builder loading a pretrained ResNet18 — runs once at startup, predicts per-upload at request time
Let's break it on stage
Login → upload real sneaker photos → watch Step 0 / 1 / 2 run live → check the admin dashboard
Bug #1 — Gemini returned an array, not an object
What happened
Uploaded photos contained two visually distinct sneakers. Gemini "helpfully" described both — as a JSON array of two identity objects. Our schema expected exactly one. Jackson threw MismatchedInputException and the whole analysis crashed.
Fix: explicit prompt instruction ("always one object, never an array") + defensive code that unwraps an array and takes the first element rather than crashing.
Bug #2 — refresh kept re-running the whole analysis
What happened
The POST handler returned the result view directly. Browser's last action stayed a POST — hitting refresh (or even our own "reload" button) silently resubmitted the form, re-running both Gemini calls.
Fix: classic POST → Redirect → GET. Then went further: AJAX fetch() submission so the page never navigates at all — uploaded photos survive across errors and re-analysis too.
Bug #3 — the model underneath you can just... change
What happened
gemini-2.0-flash started returning 404 — model no longer available mid-build, with no code change on our side. A different model also briefly hit a free-tier quota of limit: 0.
This isn't a Gemini-specific quirk — it's the industry norm. OpenAI deprecates and retires model snapshots constantly (gpt-4-0314, gpt-4-32k, multiple preview tags gone within months). Anthropic and Google do the same on a slower cadence.
Fix: moved to gemini-2.5-flash, added retry-with-exponential-backoff for transient 503/429s, and — the real lesson — never hardcode a model ID as a fact you can forget about. Treat it as a config value you'll revisit, the same way you'd treat a third-party API version string.
What does one analysis actually cost?
Runs on your own CPU. No API cost, ~0.2-0.7s.
tokens (input-heavy — images dominate the cost)
tokens (more output — evidence, verdict, resale)
Shown live in the app's own "Agent Token Usage" panel — built so the audience can see the real cost, not a guess.
How does an image become "tokens"?
- →Gemini tiles each image into fixed chunks — roughly 258 tokens per tile for images ≤384×384; bigger photos get split into multiple tiles. A typical phone photo can run 1-2 tiles.
- →Today, SneakerLens sends raw bytes inline via
Part.fromBytes()— JSON over HTTP requires this to travel as base64 on the wire (~33% size inflation over raw bytes). - →The expensive mistake: we send all 4 photos in Step 1, then send the same 4 photos again in Step 2 — full image-token cost paid twice for identical bytes.
The fix we didn't ship (yet)
Gemini's Files API lets you upload media once, get back a file reference, and pass that reference into multiple generateContent calls (valid 48h) — pay the upload cost once, not per step.
Inline bytes vs. URI reference — does it matter?
| Approach | How it works | Tradeoff |
|---|---|---|
| Inline base64 (what we do today) | Image bytes embedded directly in the JSON request body | Simple, but re-sent on every call — bigger payloads, more latency, no reuse |
| Files API reference (Gemini Developer API) | Upload once → get a file:// handle → reference it in N calls | Upload cost paid once across our 2-step chain; smaller request bodies |
| GCS URI (Vertex AI only) | Point Gemini at gs://bucket/photo.jpg directly | No re-upload at all if the file already lives in GCS; needs Vertex AI, not the plain API key |
| S3 URI | Not natively supported by Gemini | Google's stack only understands GCS — an S3 file has to be copied/proxied first |
For a 2-step agent chain reusing the same photos, Files API is the easy win — it's a same-vendor, same-API-key change.
What this looks like as a real GCP deployment
Key change from today's demo: Redis-backed sessions — our current HttpSession lives in one JVM's memory and breaks the moment Cloud Run scales past 1 instance.
Encryption: at rest vs. in transit
🔒 In transit
Data moving between systems — browser↔LB, LB↔Cloud Run, Cloud Run↔Cloud SQL, Cloud Run↔Gemini.
- TLS everywhere, no exceptions — including internal hops
- Cloud SQL: enforce SSL or use the Cloud SQL Auth Proxy (mTLS, no manual cert management)
- Gemini calls already go over HTTPS by default — nothing extra to configure
🗄️ At rest
Data sitting still — on disk in Cloud SQL, objects in GCS, Redis snapshots.
- Google-managed encryption is on by default for Cloud SQL, GCS, Memorystore — no extra setup
- Stricter compliance (e.g. PII, finance)? Use CMEK — Customer-Managed Encryption Keys via Cloud KMS, so you control key rotation/revocation
- Sneaker photos are low-sensitivity, but user phone numbers in
userstable are exactly the kind of field CMEK is meant for
Rule of thumb: in transit is about who can intercept; at rest is about who can read the disk if they get physical/storage-level access.
What if the data going to the LLM is encrypted?
The hard constraint: LLMs need plaintext
There's no shortcut here — Gemini (or any model) cannot reason over ciphertext. Homomorphic encryption + LLM inference is still research, not something you ship. If the model needs to "see" the data, it has to be decrypted first.
- →So where does decryption happen? Inside Cloud Run, in memory, right before the Gemini call — using a key pulled from Cloud KMS / Secret Manager. Never decrypt earlier, never persist the plaintext, never log it.
- →That makes the LLM call itself a deliberate decryption boundary — not a side effect you discover later in a security review.
- →Important caveat: TLS only protects data in transit to Gemini's servers — once it arrives, Google's infrastructure sees plaintext to actually run inference. That's a different trust boundary than "encrypted everywhere," and worth calling out explicitly to stakeholders.
If true confidentiality is non-negotiable
Minimize what you send (crop/redact anything not needed for the task), use enterprise terms with data-retention guarantees (Vertex AI vs. default consumer API), or self-host an open model inside your own VPC so plaintext never leaves your trust boundary at all.
Three things to remember
- 1"Agent" = output reshapes the next step. Don't reach for agent frameworks when a single call or a fixed pipeline does the job.
- 2Hybrid local+cloud is a cost pattern, not a buzzword. A cheap pretrained classifier can save real money by gating an expensive multimodal call.
- 3Java is a perfectly good place to build this. Spring AI + the official Gemini SDK + DJL — no Python required anywhere in this stack.