My personal AI benchmark: "Generate an SVG of a frog with a Habsburg jaw."

The secret to great interview questions and challenge tests is keeping them secret. Posting them on HN and getting them onto the front page puts them in jeopardy.

I thought this was great, and hilarious. Kudos to Opus 5, I thought it was the only one that came close to passing. Interestingly, I thought many of the failures drew the frog face OK, and they had some type of big blob for the jaw, so they knew "Hapsburg jaw" meant a protruding jaw, but it wasn't really connected to the frog face in any way that made sense.

Small side note, the first gemini-2.5-pro one totally reminded me of some sad faced meme or Pepe the frog from somewhere. Anyone know what I'm referring to, tried to find it.

Hi all, the site is getting hugged to death, thank you, was not expecting this kind of warm response. I will be working to make this more reliable, in the meantime, sign up for my newsletter: https://www.jaymollica.com/blog/

also my favorite SVG was def the google/gemini-3.6-flash

edit: ok better now I think

No gpt-5.6 sol and no fable?

Check out my MacBook SVG benchmark. From my experience, it demonstrates the Real model’s behavior. However, I notice the errors it makes, which are similar to the mistakes made by the mistake model in code.

https://playcode.io/blog/macbook-svg-benchmark

Opus 5 clearly frogmaxxed.

gemini-3.6-flash runs 2 and 3 responded best to the royal portrait context.

Here is GLM 5.2 (https://codeinput.com/s/HAO0qTxw2ia) which is still inferior to Opus. I can't find Qwen 3.8 which now is my daily driver replacing GLM. This SVG test matches my experience when working with the different models. The other models can get the details right but their output is structured in a way that makes little or no sense.

I also did a timeline from 4.7 to 5.2: https://codeinput.com/s/7oK2IIA7qRO The improvements in models looks much less impressive with this test.

Gemini 2.5 Pro fails, but has a distinctive art style that is quite nice. It seems to understand shading to a much higher level than all other models.

My personal human benchmark: "Jump on one leg, while reciting the national anthem of Latvia, translated to Spanish, backwards, while drawing a frog with a brush held by toes of the other leg, on the ceiling". So far they're not doing very good but I'm sure they'll improve over time.

Please could we have a human generated image to compare the AI generated tosh with?

It's opus 5 > Kimi K3 > grok 4.5

That's a pretty good benchmark

This is a strong benchmark! None of these could be remotely mistaken for human art. Opus 5 comes closest.

How do models approach SVG generation? In one version, I imagine them actually trying to reason about them as an LLM. In another, I imagine something closer to a GAN.

My test is to ask AI to pick up all the rubbish at the beach.

A friend’s favorite prompt is “Batman & Julia Child; in the kitchen laughing at a ham”. Sounds simple, but has been surprisingly tough.

Can you also try the new deepseek v4 flash?

I don't get the point of these benchmarks, what are they supposed to represent practically?

Gemini 3.6 flash is crazy good.

Would've wanted to see also DS4 flash.

Mine is any variations on mammoths in various situations, or anthropomorphic. Since mammoths are invariably majestically going from one place to another in any of the books, models have hard time imagining anything but that.

Also try a fantasy archer with a proper bow who is not brooding, sitting in a fantasy wood :)

I think this one has advantages over the “pelican riding a bicycle” one because it hinges on an anatomical feature that many models associate with royalty, “habsburg” being a lineage and “habsburg jaw” being an anatomical feature.

Seven of fourteen models silently imported royalty into a prompt that named only an anatomical feature. Two of them knew they were extrapolating ("because Habsburg") and did it anyway.

Mistral returned byte-identical output across separate calls.

Gemini narrates its work in 65 comments; Llama says nothing.

If you're deciding which model to trust with instructions, "how much does it embellish beyond what I asked" and "does it behave deterministically" are directly practical questions.

For those who don’t know a Habsburg jaw also known as mandibular prognathism, it is a genetic condition characterized by a protruding lower jaw, which was notably prevalent among members of the Habsburg royal family due to their history of inbreeding. This condition often resulted in significant facial deformities and difficulties with eating and speaking.

The secret to great interview questions and challenge tests is keeping them secret. Posting them on HN and getting them onto the front page puts them in jeopardy.

I also did a timeline from 4.7 to 5.2: https://codeinput.com/s/7oK2IIA7qRO The improvements in models looks much less impressive with this test.

https://playcode.io/blog/macbook-svg-benchmark

It's opus 5 > Kimi K3 > grok 4.5

That's a pretty good benchmark

No gpt-5.6 sol and no fable?

This is a strong benchmark! None of these could be remotely mistaken for human art. Opus 5 comes closest.

Please could we have a human generated image to compare the AI generated tosh with?

Gemini 2.5 Pro fails, but has a distinctive art style that is quite nice. It seems to understand shading to a much higher level than all other models.

How do models approach SVG generation? In one version, I imagine them actually trying to reason about them as an LLM. In another, I imagine something closer to a GAN.

Also try a fantasy archer with a proper bow who is not brooding, sitting in a fantasy wood :)

Small side note, the first gemini-2.5-pro one totally reminded me of some sad faced meme or Pepe the frog from somewhere. Anyone know what I'm referring to, tried to find it.

Perhaps you're thinking of Salad Fingers?

Gemini 3.6 flash is crazy good.

Would've wanted to see also DS4 flash.

Crazy funny, yes, but not good.

My test is to ask AI to pick up all the rubbish at the beach.

Opus 5 clearly frogmaxxed.

gemini-3.6-flash runs 2 and 3 responded best to the royal portrait context.

Arguably, a royal portrait is a misinterpretation of the prompt, since it's just asking for a specific facial feature. But I guess you could look at it as a bit of artistic license.

> frogmaxxed

raninemandibularprognathism-maxxed?

also my favorite SVG was def the google/gemini-3.6-flash

edit: ok better now I think

> also my favorite SVG was def the google/gemini-3.6-flash

That looks like something from Machinarium or Robots :)

I don't get the point of these benchmarks, what are they supposed to represent practically?

The ability of an LLM to produce something not in its training data set.

For me this looks ideological (or even political), not practical. The theory is that LLMs are approaching general intelligence (whatever that means) and that the more generic of a task they can perform—no matter how badly—the closer we are to AGI.

Specialized models can do this a lot better and for far cheaper then LLMs, but because people are so politically invested in a single statistical model being able to outperform a human on every metric (no matter how expensive the compute), then we get these ridiculous benchmarks.

Can you also try the new deepseek v4 flash?

I will add it to next month's report!

Seven of fourteen models silently imported royalty into a prompt that named only an anatomical feature. Two of them knew they were extrapolating ("because Habsburg") and did it anyway.

Mistral returned byte-identical output across separate calls.

Gemini narrates its work in 65 comments; Llama says nothing.

If you're deciding which model to trust with instructions, "how much does it embellish beyond what I asked" and "does it behave deterministically" are directly practical questions.

The identical pair from Mistral took me off guard. Many of the other models were so varied between the runs which is more what I would expect.

A friend’s favorite prompt is “Batman & Julia Child; in the kitchen laughing at a ham”. Sounds simple, but has been surprisingly tough.

Perhaps you're thinking of Salad Fingers?

~I'm leaning more towards the rage face poker face~

edit: nevermind, definitely "monkey-puppet side-eye" vibe.

> also my favorite SVG was def the google/gemini-3.6-flash

That looks like something from Machinarium or Robots :)

> frogmaxxed

raninemandibularprognathism-maxxed?

Crazy funny, yes, but not good.

I think that on the rendering side, it's miles ahead of the rest, even if off topic.

The ability of an LLM to produce something not in its training data set.

Arguably, a royal portrait is a misinterpretation of the prompt, since it's just asking for a specific facial feature. But I guess you could look at it as a bit of artistic license.

~I'm leaning more towards the rage face poker face~

edit: nevermind, definitely "monkey-puppet side-eye" vibe.

I will add it to next month's report!

clearly I missed an amazing copy opportunity, thank you haha

The identical pair from Mistral took me off guard. Many of the other models were so varied between the runs which is more what I would expect.

I wonder if the setup accidentally hit a cache at some layer.

I'm assuming they mean an SVG.

I think that on the rendering side, it's miles ahead of the rest, even if off topic.

I wonder if the setup accidentally hit a cache at some layer.

I'm assuming they mean an SVG.

clearly I missed an amazing copy opportunity, thank you haha

The model added significant editorializing beyond a literal depiction, including invented royal status ("Royal Imperial Collar," "Imperial Habsburg Crown," "Golden Fleece Medal") and mood/bearing descriptors like "Folded Pompously," "gloomy expression," and "Sad/Weary facial lines." It also inserted anatomical commentary framing features as medical/dynastic traits, such as "Belly Plate (Weak chest)," "Heavy Droopy Eyelids (Habsburg lethargic look)," and "weak maxilla."

View SVG source

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 600 600" width="100%" height="100%">
  <defs>
    <!-- Skin Gradients -->
    <radialGradient id="frogSkin" cx="40%" cy="30%" r="70%">
      <stop offset="0%" stop-color="#7bc65d"/>
      <stop offset="60%" stop-color="#478c33"/>
      <stop offset="100%" stop-color="#255218"/>
    </radialGradient>
    
    <radialGradient id="jawSkin" cx="30%" cy="30%" r="70%">
      <stop offset="0%" stop-color="#8ede6c"/>
      <stop offset="50%" stop-color="#529e3a"/>
      <stop offset="100%" stop-color="#2d5e1e"/>
    </radialGradient>

    <linearGradient id="bellySkin" x1="0%" y1="0%" x2="100%" y2="100%">
      <stop offset="0%" stop-color="#e2f7b3"/>
      <stop offset="70%" stop-color="#b6e071"/>
      <stop offset="100%" stop-color="#80ab3e"/>
    </linearGradient>

    <!-- Eye Gradient -->
    <radialGradient id="eyeIris" cx="50%" cy="50%" r="50%">
      <stop offset="0%" stop-color="#ffd700"/>
      <stop offset="70%" stop-color="#da9100"/>
      <stop offset="100%" stop-color="#804000"/>
    </radialGradient>

    <!-- Crown Gradient -->
    <linearGradient id="gold" x1="0%" y1="0%" x2="100%" y2="100%">
      <stop offset="0%" stop-color="#ffeea1"/>
      <stop offset="40%" stop-color="#d4af37"/>
      <stop offset="100%" stop-color="#AA771C"/>
    </linearGradient>

    <!-- Velvet/Ruff Gradients -->
    <linearGradient id="ruff" x1="0%" y1="0%" x2="0%" y2="100%">
      <stop offset="0%" stop-color="#ffffff"/>
      <stop offset="70%" stop-color="#e0e0e0"/>
      <stop offset="100%" stop-color="#a0a0a0"/>
    </linearGradient>

    <radialGradient id="ruby" cx="35%" cy="35%" r="60%">
      <stop offset="0%" stop-color="#ff9999"/>
      <stop offset="30%" stop-color="#cc0000"/>
      <stop offset="100%" stop-color="#4a0000"/>
    </radialGradient>

    <!-- Shadows -->
    <radialGradient id="dropShadow" cx="50%" cy="50%" r="50%">
      <stop offset="0%" stop-color="rgba(0,0,0,0.4)"/>
      <stop offset="100%" stop-color="rgba(0,0,0,0)"/>
    </radialGradient>

    <filter id="shadowFilter" x="-10%" y="-10%" width="120%" height="120%">
      <feDropShadow dx="2" dy="4" stdDeviation="4" flood-opacity="0.3"/>
    </filter>
  </defs>

  <!-- Background Shadow -->
  <ellipse cx="300" cy="530" rx="180" ry="25" fill="url(#dropShadow)"/>

  <!-- Back Body & Legs -->
  <g id="body-and-legs">
    <!-- Left/Back Leg -->
    <path d="M 180 380 C 120 390, 90 450, 130 510 C 150 540, 200 530, 210 500 C 180 490, 160 450, 200 410 Z" fill="#356625"/>
    
    <!-- Right/Main Back Leg -->
    <path d="M 360 360 C 450 370, 500 440, 460 510 C 430 550, 370 540, 350 510 C 400 500, 420 440, 360 400 Z" fill="url(#frogSkin)"/>
    <path d="M 420 500 L 460 530 M 400 510 L 420 538 M 380 512 L 380 540" stroke="#255218" stroke-width="5" stroke-linecap="round"/>

    <!-- Main Torso -->
    <path d="M 200 320 C 170 380, 200 490, 270 510 C 340 520, 400 460, 380 350 C 340 310, 240 300, 200 320 Z" fill="url(#frogSkin)"/>
    
    <!-- Belly Plate (Weak chest) -->
    <path d="M 220 330 C 200 380, 220 470, 280 490 C 330 500, 360 450, 350 360 C 300 320, 250 315, 220 330 Z" fill="url(#bellySkin)"/>
  </g>

  <!-- Royal Imperial Collar (Ruff) to frame the jaw -->
  <g id="ruff-collar" filter="url(#shadowFilter)">
    <path d="M 190 320 
             Q 170 300, 195 285 Q 170 270, 200 260 Q 185 240, 220 240 
             Q 220 220, 250 230 Q 260 210, 290 225 Q 310 210, 330 230 
             Q 350 220, 365 245 Q 390 245, 385 270 Q 410 280, 390 300 
             Q 410 320, 380 330 C 330 370, 230 360, 190 320 Z" fill="url(#ruff)" stroke="#d0d0d0" stroke-width="2"/>
    <!-- Ruff folds -->
    <path d="M 200 260 C 230 280, 250 310, 250 330 M 220 240 C 240 270, 270 310, 280 335 M 250 230 C 270 260, 300 310, 310 335 M 290 225 C 300 260, 330 300, 340 330 M 330 230 C 330 260, 350 290, 365 315" stroke="#b0b0b0" stroke-width="1.5" fill="none"/>
  </g>

  <!-- Head Base & Eyes -->
  <g id="head-back">
    <!-- Eyeball Tubercles -->
    <circle cx="230" cy="150" r="45" fill="url(#frogSkin)"/>
    <circle cx="335" cy="165" r="38" fill="url(#frogSkin)"/>
  </g>

  <!-- Eyes Outer Details -->
  <g id="eyes">
    <!-- Left Eye (Viewer's Left) -->
    <circle cx="235" cy="150" r="32" fill="#1a330e"/>
    <circle cx="235" cy="150" r="28" fill="url(#eyeIris)"/>
    <!-- Horizontal Frog Pupil (Droopy/Tired) -->
    <ellipse cx="235" cy="150" rx="18" ry="6" fill="#050a03" transform="rotate(-5, 235, 150)"/>
    <!-- Eye Highlight -->
    <circle cx="225" cy="140" r="4" fill="#ffffff" opacity="0.8"/>
    <circle cx="220" cy="146" r="2" fill="#ffffff" opacity="0.6"/>

    <!-- Right Eye (Viewer's Right) -->
    <circle cx="335" cy="165" r="26" fill="#1a330e"/>
    <circle cx="335" cy="165" r="22" fill="url(#eyeIris)"/>
    <!-- Horizontal Pupil -->
    <ellipse cx="335" cy="165" rx="14" ry="5" fill="#050a03" transform="rotate(10, 335, 165)"/>
    <!-- Eye Highlight -->
    <circle cx="328" cy="158" r="3" fill="#ffffff" opacity="0.8"/>

    <!-- Heavy Droopy Eyelids (Habsburg lethargic look) -->
    <path d="M 190 140 Q 235 115, 278 142 Q 250 145, 190 140 Z" fill="url(#frogSkin)"/>
    <path d="M 190 140 Q 235 115, 278 142" stroke="#255218" stroke-width="3" fill="none"/>
    
    <path d="M 300 158 Q 335 138, 370 160 Q 340 162, 300 158 Z" fill="url(#frogSkin)"/>
    <path d="M 300 158 Q 335 138, 370 160" stroke="#255218" stroke-width="3" fill="none"/>
  </g>

  <!-- Weak Upper Snout / Nose -->
  <g id="upper-head">
    <!-- Small, slightly sunken upper face -->
    <path d="M 210 170 C 210 200, 230 220, 260 225 C 280 228, 320 220, 340 185 C 320 180, 230 165, 210 170 Z" fill="url(#frogSkin)"/>
    <!-- Nostrils (Very close together, weak maxilla) -->
    <ellipse cx="245" cy="205" rx="2" ry="4" fill="#15300d" transform="rotate(-10, 245, 205)"/>
    <ellipse cx="258" cy="207" rx="2" ry="4" fill="#15300d" transform="rotate(5, 258, 207)"/>
  </g>

  <!-- THE HABSBURG JAW (Massive Protruding Mandible) -->
  <g id="habsburg-jaw" filter="url(#shadowFilter)">
    <!-- Base massive jaw structure jutting FAR forward and down -->
    <!-- Starts from ear area, sweeps way out past upper snout, forms huge chin, comes back -->
    <path d="M 175 250 
             C 140 255, 110 280, 120 310 
             C 125 330, 140 340, 160 365 
             C 185 395, 210 415, 265 420 
             C 320 425, 360 395, 375 350 
             C 385 320, 370 280, 345 260 
             C 310 285, 260 290, 220 280 
             C 195 273, 185 260, 175 250 Z" fill="url(#jawSkin)"/>

    <!-- Under-jaw / Distance depth -->
    <path d="M 160 365 C 190 420, 240 435, 285 430 C 340 425, 370 380, 375 350 C 350 390, 300 415, 260 410 C 205 405, 175 380, 160 365 Z" fill="#1f4213"/>

    <!-- Pouting Everted Lower Lip (Classic Habsburg Feature) -->
    <path d="M 170 260 
             Q 130 275, 135 295 
             Q 145 320, 180 325 
             Q 230 330, 280 315 
             Q 330 300, 350 270 
             Q 300 275, 250 270 
             Q 200 265, 170 260 Z" fill="#6ba64e" stroke="#2d5e1e" stroke-width="2"/>
             
    <!-- Inner Lip / Mouth Line (Downturned, gloomy expression) -->
    <path d="M 175 262 Q 220 280, 270 275 Q 320 270, 345 262" stroke="#102409" stroke-width="4" stroke-linecap="round" fill="none"/>
    
    <!-- Lip Highlight (Moist lower lip) -->
    <path d="M 150 295 Q 175 315, 220 318" stroke="#a2f081" stroke-width="3" stroke-linecap="round" fill="none" opacity="0.6"/>

    <!-- Prominent Chin Crease / Unfused Mandible Indentation -->
    <path d="M 235 345 C 240 375, 250 395, 255 415" stroke="#255218" stroke-width="3" stroke-linecap="round" fill="none"/>
    <path d="M 230 345 C 235 375, 243 395, 247 414" stroke="#7bc65d" stroke-width="1.5" fill="none"/>

    <!-- Chin Shadow/Highlight defining the extreme protrusion -->
    <path d="M 180 330 C 200 370, 230 390, 260 390 C 300 390, 340 360, 355 320" stroke="#387027" stroke-width="3" fill="none"/>
    <circle cx="215" cy="365" r="25" fill="#ffffff" opacity="0.08"/>
  </g>

  <!-- Front Arms / Paws (Folded Pompously over belly) -->
  <g id="arms">
    <!-- Left Arm -->
    <path d="M 190 320 C 160 340, 150 380, 180 410 C 190 420, 210 410, 220 390 C 200 380, 195 350, 205 330 Z" fill="url(#frogSkin)"/>
    <!-- Left Hand resting on chin/chest -->
    <path d="M 180 410 Q 200 425, 220 400" stroke="#255218" stroke-width="3" fill="none"/>

    <!-- Right Arm (Slightly hidden, hand under the massive chin) -->
    <path d="M 360 320 C 380 340, 390 380, 360 410 C 345 420, 325 400, 325 385 C 340 375, 345 350, 340 330 Z" fill="url(#frogSkin)"/>
  </g>

  <!-- Imperial Habsburg Crown (Sits awkwardly small between large eye sockets) -->
  <g id="crown" filter="url(#shadowFilter)">
    <!-- Crown Base -->
    <path d="M 260 125 L 310 133 L 305 143 L 255 135 Z" fill="url(#gold)"/>
    
    <!-- Crown Spikes / Arch -->
    <path d="M 255 135 
             L 245 105 L 268 120 
             L 280 95 L 292 123 
             L 315 110 L 305 143 Z" fill="url(#gold)" stroke="#8a5d0f" stroke-width="1"/>

    <!-- Imperial Arch (Mitre style Habsburg Crown element) -->
    <path d="M 265 120 Q 280 90, 295 100 Q 280 105, 265 120 Z" fill="#990000"/>
    <path d="M 280 95 Q 283 85, 280 80" stroke="url(#gold)" stroke-width="3" fill="none"/>
    <!-- Cross on top -->
    <path d="M 276 80 L 284 80 M 280 76 L 280 84" stroke="#ffeea1" stroke-width="2"/>

    <!-- Jewels -->
    <circle cx="280" cy="95" r="2.5" fill="url(#ruby)"/>
    <circle cx="245" cy="105" r="2" fill="url(#ruby)"/>
    <circle cx="315" cy="110" r="2" fill="url(#ruby)"/>
    <circle cx="270" cy="131" r="1.5" fill="#ffffff"/>
    <circle cx="295" cy="135" r="1.5" fill="#ffffff"/>
  </g>

  <!-- Skin Texture / Habsburg Flaws (Warts & Spots) -->
  <g id="details" opacity="0.4">
    <!-- Frog Warts -->
    <circle cx="150" cy="330" r="4" fill="#15300d"/>
    <circle cx="142" cy="320" r="2.5" fill="#15300d"/>
    <circle cx="330" cy="340" r="5" fill="#15300d"/>
    <circle cx="345" cy="350" r="3" fill="#15300d"/>
    <circle cx="280" cy="380" r="4" fill="#15300d"/>
    <circle cx="295" cy="390" r="2.5" fill="#15300d"/>
    <circle cx="210" cy="370" r="3.5" fill="#15300d"/>
    
    <!-- Sad/Weary facial lines -->
    <path d="M 225 180 Q 215 195, 220 210" stroke="#15300d" stroke-width="1.5" fill="none"/>
    <path d="M 315 185 Q 320 195, 315 210" stroke="#15300d" stroke-width="1.5" fill="none"/>
  </g>

  <!-- Golden Fleece Medal (Imperial Habsburg Order) hanging around neck -->
  <g id="golden-fleece" filter="url(#shadowFilter)">
    <path d="M 250 335 Q 270 360, 285 355" stroke="#d4af37" stroke-width="3" fill="none" stroke-dasharray="4,2"/>
    <!-- Small sheep/fleece oval element -->
    <ellipse cx="288" cy="360" rx="7" ry="5" fill="url(#gold)"/>
    <circle cx="288" cy="354" r="2" fill="url(#gold)"/>
  </g>
</svg>

Hacker Times

Hacker Times

My personal AI benchmark: "Generate an SVG of a frog with a Habsburg jaw."

Discussion

Discussion