EcoRetrofit AI is a multi-modal home energy assistant built with Google AI Studio to enable users to do energy audits and optimise energy usage and efficiency based on energy and structural data compared to other home profiles.

Built with React, Vite, and Gemini Pro 3, it empowers users to easily perform a comprehensive assessment of their home, receiving personalised recommendations on how to reduce or optimise usage, improve energy efficiency, and ultimately reduce their carbon footprint.

1 Reason & Motivation

Navigating the world of energy retrofits can be incredibly confusing and intimidating, especially for homeowners without an engineering background. The process is often opaque: every building is unique, and the financial payback of any upgrade hinges on a complex web of variables including local climate data, contractor availability, and ever-changing government incentives.

Eco Retrofit AI was designed to help analyse pdf bills, multi-modal home walkthrough videos, images, and documents—and frame technical upgrades in plain, accessible language. The goal is to make climate action feel actionable and personal, rather than overwhelming and abstract.

2 How it Works

Users first upload their historical energy bills (PDFs), a quick walkthrough video of their home, and any specific pictures of interest (like a draughty window, old furnace, boiler installation or smart meter installation).

Under the hood, Gemini 2.5 Flash is used to do the core “Energy Auditor” logic due to its speed and reasoning capabilities. It distinguishes between Renters and Homeowners based on the user’s situation, dynamically adjusting its system prompts to ensure recommendations are appropriate (e.g., suggesting non-structural fixes like draught excluders for renters vs. heat pumps for owners).

Once the data is ingested, the system presents a dashboard of recommended actions, ranked by potential impact. Users can see:

Projected Annual Savings: Estimates based on specific consumption patterns.
Upfront Investment & Payback Period: Clear financial metrics to help prioritise.
Visual Analysis: Insights derived directly from the video walkthrough. To handle large video files efficiently on the client side, representative frames are extracted using a canvas element with a safety timeout to prevent browser hangs.
Comparative Predictions: Benchmarks against other homes with similar profiles to contextualise usage.

The system also estimates the home’s current energy efficiency rating. If there is an official Energy Performance Certificate (EPC), the system extracts the data; otherwise, it estimates the rating based on visual evidence (like glazing type and insulation thickness) found in the photos.

A Chat Interface (EcoRetrofit Copilot) is also included, allowing users to ask follow-up questions about the analysis. It uses the full context of the audit—including usage data, identified inefficiencies, and recommended upgrades—to provide specific answers, acting as an on-demand energy consultant.

One of the most powerful features is the Generative 3D Visualisation. The app uses Gemini 3 Pro (gemini-3-pro-image-preview) to “dream” up 3D cutaways of the home in various styles (Realistic, Clay, Blueprint). For the interactive demo, a custom “Mini 3D Engine” was built using Three.js and HTML5 Canvas, where elements were manually “vibe-coded”—even using cosine functions to mathematically drape fairy lights (y = height + Math.cos(i * 0.5) * 8) to make the space feel warm and lived-in.

The analysis is dynamic and stored locally using JS IndexedDB API. If something is missing, users can simply add new information—for example, uploading a new video of the attic—and the system will update its recommendations and save a new version of the audit.

3 Potential Features

The roadmap for future development includes:

Advanced Multi-modal Ingestion: Expanding support to include thermal imaging scans, architectural floor plans, and unstructured occupant notes to build a richer context.
Financial Projection Module: A robust financial layer integrated with real-time rebate APIs (eg. local utility databases) to provide accurate, location-specific cost-benefit analyses.
Seasonal Simulation Snapshots: Using typical meteorological year data to simulate how the home performs across different seasons, helping users prepare for peak energy loads.
“Retrofit Bundle” Templates: Pre-packaged sets of upgrades (eg. “Weatherisation Starter Pack”, “Solar & Battery Combo”) to streamline the quoting and decision-making process to suggestion

4 Challenges

Building this system wasn’t without its hurdles:

Multimodal Hallucinations: Early prototypes occasionally “invented” rebates or technologies that didn’t exist. To fix this, strict grounding techniques were implemented by injecting a Verified Sources Library (containing URLs from the Energy Saving Trust) directly into the system prompt. The model is strictly instructed to cite only these sources, effectively clamping its outputs to a curated fact bank.
Performance Optimisation: While Vertex AI and Gemini streaming are fast, analysing high-resolution images and videos can spike cold start times. A fallback mechanism was implemented where if the high-fidelity Gemini 3 Pro fails to generate a visualisation, the system automatically retries with the faster Gemini 2.5 Flash Image model to ensure the user flow isn’t interrupted.
Code Maintainability/Readability: Being new to the vibe coding world and also ReAct web-apps, it seems the generated code was not good at more elegantly abstracting away and modularising the styling CSS code etc from the layout logic. Further experimentation needs to be done to make this repo actually maintainable/readable beyond this initial AI studio vibe code session! Same goes with the commit history.
Change Reversion: There were occasional times when introducing a new change would revert old features, perhaps there was context rot or limitation in the context handling after a session is sufficiently long.
Non-complete System Features: Perhaps this may be due to the way I am prompting Gemini where I occasionally don’t make instructions detailed, step-by-step, but this often meant that changes were done only at the surface-level and not in a way that makes sense throughout the code. An example of this is when Gemini was not syncing session history changes in “Demo Data” and the “History” panel with versions without explicit corrections. These would not happen until had to explicitly set a loadHistory function that would load all historical items from the database and saveAnalysis to handle the “Update” logic.

5 Learnings

AI Studio Iteration Speed: Google AI Studio proved to be a game-changer with “vibe coding.” The ability to rapidly swap prompts, verify JSON schemas, and see the flow live significantly accelerated the development cycle.
Role-Playing Prompts: Defining specific personas in the prompt—such as “You are a world-class AI Architect” or “You are an expert Home Energy Auditor”—significantly improved the quality and tone of the output.
Guardrails for Multimodal Prompts: Multimodal models need explicit instructions to handle uncertainty. It’s crucial to instruct the model to acknowledge when data is insufficient and to nudge the user to gather more inputs, rather than guessing.
Transparency Builds Trust: Users are more likely to act on advice they understand. Pairing each recommendation with a confidence score, a clear payback range, and links to reputable sources was essential for building trust in the AI’s suggestions.

6 Summary

Eco Retrofit AI brings together the power of Google AI Studio, Gemini’s multimodal APIs, and a comprehensive dataset of building envelopes. It bridges the gap between complex building analysis and homeowner action, helping users reason over photos, floor plans, and utility bills.

Creating this in Google AI Studio makes it super easy to build protototype/MVP and build/iterate on ideas quickly. However, it can lead to very verbose, unreadable and overly complex code if not well structured early and definitely not suitably for anything beyond a pet project and production involving maintainers other than yourself!

Open in Github