Back

One Village Weather Forecast Transformer

Training small weather Transformers for one village on the Breton coast. A rabbit hole through radar archives, atmospheric data, and what it means to make a forecast that actually works outside a notebook.

Notes

I wanted to know whether tomorrow would be sunny on the beach in Kerlouan. Not somewhere around Brest. Kerlouan, its village, its coast.

That became a fairly consuming question: could a small model, trained specifically for one place, learn something that a general-purpose forecast misses? The ambition is narrow. Predict the next 24 hours, especially rain and cloud cover, and eventually find the situations where local specialization earns its keep. Beating a national weather service is still a question, not a result.

Most of the work was finding the right data

I started with ERA5, a historical reconstruction of the atmosphere, and experimented with surface fields, humidity, wind, and pressure levels. But an excellent historical dataset is not necessarily an input you can get when you want to issue a forecast. The model has to train on information that could actually have been available at that moment.

That constraint changed the project. The newer recipe uses past and present GFS analyses for the atmospheric context, with local radar and satellite data as the finer-scale sources for rain and clouds. Future GFS analyses can provide training targets; future numerical forecasts do not go into the independent model.

There is a lot hidden behind the word "dataset": extracting small geographic regions from huge files, keeping the real timestamps, tracking missing periods, checking units, and making interrupted downloads resumable. I built collection and extraction pipelines on a Contabo server, stored compact datasets on Cloudflare R2, and used RunPod GPUs for the larger training runs. Matching the historical sources to what can be collected live became as important as the network itself.

Giving attention a place and a time

I explored Transformers with local, regional, and synoptic context, surface and altitude fields, and explicit spatial and temporal relationships. Some versions used factorized 4D attention. Others added radar history, dense training objectives, or separate heads for different weather variables.

I also tried combining the local model with future IFS forecasts. That branch improved several measured scores, but it answered a different question: how to correct an existing forecast. I kept it separate from the attempt to build an independent forecaster.

After a lot of experiments, I deliberately simplified. The latest small baseline takes four consecutive GFS analyses, six hours apart, and predicts the next four. It keeps atmospheric levels in the input but trains against surface pressure, temperature, dew point, and the two wind components. This stripped-down experiment does not yet predict local rain or cloud cover.

It has about 438,000 parameters. Historical map tokens and future date queries go through one attention stack. In a matched test, making that attention causal by time blocks did not improve the result. A more elaborate architecture is not automatically a better weather model.

What has actually worked

On a reused 2023 development set, extending one surface-only model from 1,000 to 5,000 training updates reduced its aggregate normalized squared error by about 10%. It beat persistence, the baseline that repeats the latest atmospheric state, by about 41% on the same measure.

Those are useful learning signals, not "41% better weather forecasts." The targets are GFS analyses, the score combines several surface variables, and this was development evaluation rather than a fresh prospective test. Wind remained difficult: the longer run got the exact one-of-eight direction sector right about 51% of the time across its four forecast horizons, and still underestimated speed.

I then connected the model to actual data receipts. A frozen September snapshot used four downloaded GFS analyses and produced four genuinely future predictions at native six-hourly timestamps. The pipeline ran end to end, but that alone does not establish forecast accuracy, and four timestamps are not an hourly forecast.

A research project, still

Codex helped write the pipelines, run controlled comparisons, and keep a research journal and experiment kanban. The useful habit was to preserve the failed experiments too: compare against persistence, replace context with unrelated history, check that future values cannot leak into the inputs, and verify that a resumed training run matches an uninterrupted one.

There is a working research system here, and models that demonstrably learn. There is not yet a validated replacement for a professional local forecast. The next proof has to come from forecasts fixed before the weather happens, scored afterward, over enough different days to mean something.

I still like the original scale of the idea. A small model, for a small place, and a very ordinary question about tomorrow.