Feature Request: Consider Integrating AI for Layout Conversion

[imdoge]

Hi maintainers,

First off, thanks for the great work on the figmatocode repository! It's a very useful tool.

I've noticed that the default mode often generates layouts using position: absolute. While this can be accurate to the pixel, it often requires significant refactoring to create responsive and maintainable components, especially when dealing with complex designs. I've also experimented with the 'auto layout' mode, but I've found it doesn't always produce the desired results for many of the designs I work with. Manually adapting the absolute-positioned code involves handling numerous complex edge cases and significantly increases development complexity.

Furthermore, many standalone AI design-to-code tools, possibly relying primarily on visual analysis, often don't achieve the same pixel-level accuracy for styles compared to figmatocode's direct Figma API integration which precisely captures attributes.

After experimenting, I found a practice that works surprisingly well by combining figmatocode's precise attribute extraction (via Figma API) with the capabilities of generative AI. The workflow is essentially:

1. Take a screenshot of the target design/component in Figma.
2. Generate the code using the figmatocode extension to get the precise attribute values.
3. Provide both the screenshot and the figmatocode-generated code to an LLM (I used Claude 3.7 Sonnet for testing).
4. Use a prompt to instruct the LLM to:
   • Combine the visual information from the screenshot with the precise attributes from the original code.
   • Replace position: absolute layouts with a flexible alternative (like Flexbox).
   • Keep all other styles exactly as they were generated by figmatocode.
   • Infer approximate padding/margin/gap/space values by analyzing the left/top differences in the original absolute positioning.
   • (Optional) Supplement the prompt with additional requirements or hints, such as considering responsive design for different devices, using Tailwind CSS, adapting for Next.js, or suggesting it ignore the content of very large SVG components to keep the output concise.

The results from this hybrid approach – leveraging figmatocode's precision, visual context, and the LLM's reasoning – have been remarkably effective. Comparing it to many SaaS tools I've tried, I believe this method often yields better outcomes.

would you consider integrating an AI-powered conversion capability directly into the figmatocode repository? This could perhaps be an optional "AI Convert" mode or a post-processing step after the initial code generation.

Thank you for considering this suggestion and for your work on this valuable tool.

[bernaferrari]

I surely could, but Claude is expensive. Very expensive. Would you pay $10-15 per month for a feature like this? Or do you prefer to bring your own API? Or are you fine with a much worse model but cheaper/free?

[imdoge]

I think it would be beneficial to provide options for custom apikey and configurable provider/baseUrl/prompt, as models like Gemini 2.0/2.5 Pro offer more affordable alternatives.

Additionally, I tested some mobile UI designs, and the token usage per page ranges approximately between 5k-9k tokens (with input tokens converted into output-token equivalents, and SVG internal content removed to reduce overall length). This cost isn't very high.

Ideally, the best approach is leveraging a visual multimodal model. Combining screenshots, Figmatocode, an LLM, and clearly defined, specific prompts yields excellent results. In my view, this combination surpasses most SaaS solutions.

[bernaferrari]

At 5-10k, if you generate 10x in Claude, it would cost about $2. If there are 10 people using it in a week, it is already $20, and the plugin has thousands of people. If I ask for API key, that could surely work, but then 90% of people wouldn't use because it is too hard.

I don't want to promote my closed source competitor, but have you tried it? https://www.figma.com/community/plugin/1443774571835235184/figma-to-ai-code-by-designcode

if it is useful, I surely could do similar things

[imdoge]

I previously didn't find this extension supports custom API key, but I've tested other SaaS products.

I just spent some time testing this extension specifically, this time using GPT-4o.

It still has the same problems I mentioned initially and that are common with some SaaS solutions: insufficient pixel accuracy. For example, font-family doesn't match exactly, colors and backgrounds aren't fully consistent, and SVG elements differ(it use icon library). The AI tends to improvise too much.

With the approach I mentioned earlier, aside from some minor inconsistencies in padding/margin/gap/spacing (it's admittedly challenging to have AI calculate exact margins using absolute(left/top) positioning differences, but other plugins face the same issue—pixel-perfect adjustments would require manual tweaking), the other styles closely follow the Figma specifications.

For instance, here's a comparison of generated text classes:

AI-generated:
text-lg font-bold text-gray-800 mb-4

My approach:
text-neutral-700 text-xl font-normal font-['Sen'] capitalize mb-4

After reviewing the Figma property annotations, I believe the latter is more accurate.

[bernaferrari]

The plugin I shared uses ai via api key and has many interesting features like history of generations and preview.