Songtrail names 150 North American bird species from a photo or a few seconds of song, all of it on the phone with no network call at any point. Four models and a handful of open datasets feed into a single ranked answer. Most of the work went into what each knows that the others cannot: a photograph carries nothing about the month, an occurrence table has never heard the recording, and a weather forecast has no idea which bird you are pointing at.

Perch and BioCLIP, on the Phone

The identification is done by Perch, Google's open-source bioacoustics model, and BioCLIP, a vision model from the Imageomics Institute. Perch was trained on more than a million and a half labelled recordings covering some 15,000 species, and BioCLIP on images spanning over 450,000 taxa of the tree of life. Songtrail runs both essentially as published, resting on training sets far larger than any one project could assemble.

Getting them onto a phone was less straightforward than choosing them. Perch does not read audio directly. It first turns the sound into a spectrogram, a picture of the recording with time along one axis and pitch up the other, and it does that conversion inside the model file using TensorFlow operations iOS will not run. So I split the model where the conversion hands its picture to the part that recognizes birds, kept the recognizer, and rewrote the conversion in Swift. Most of the time went on proving my version feeds the recognizer exactly what Perch's own would have, matching its labels rather than coming close. Size was the other worry, since 130 MB of weights is a lot to hold in memory, though these proved far easier guests than the last on-device model I tried to ship. None of it is visible in the app, which is the sort of plumbing I keep running into between a prototype and a product.

Knowing There's a Bird at All

Neither species model has a class for "not a bird", so both describe an empty room or a blank wall with complete confidence, and in testing a plain white wall came back as 54% one species. Two further models exist to catch that, one watching the photos and one listening to the audio.

For photos, a COCO object detector finds the bird and crops to it, and that same detector doubles as the gate. Its confident verdicts are trusted both ways: above roughly 0.30 there is a bird in the frame, below 0.04 there is not, whatever the species model believes, which is what rejects walls, sky, and grass. Between those thresholds it is genuinely unsure, and there the species model gets the deciding vote, because a bird small in the frame or filling it entirely can score as little as 0.08 while the classifier is perfectly certain.

Audio is checked three ways. A tonality floor goes first, measuring how narrow-band the clip is, since birdsong is tonal where traffic and an empty room are flat, and anything too flat never reaches the species model. Apple's Sound Analysis classifier, which ships with iOS and tells birdsong from speech, music, and silence, then settles the obvious cases either way, precisely the distinction a 138-way species model cannot draw about itself. What survives both is judged on whether the species model agreed with itself across the recording, since a real bird gives the same answer window after window where a passing noise does not.

Where and When

Once something is confirmed to be a bird, the ranking picks up what the recording could not contain. A 0.62 MB table holds occurrence counts from GBIF, the Global Biodiversity Information Facility, for all 150 species across every US state and Canadian province, by month. The app normalizes those counts for where you are standing and the month you are in, then multiplies them into the model's output, with the table's pull deliberately damped so that it tilts a ranking without ever overruling one.

Baltimore and Bullock's Orioles resemble each other closely enough that the model splits 0.50 to 0.45 between them and declines to commit, which is right for something shown a photograph and nothing else. With the table applied, the same output resolves to Baltimore at 0.99 in New York and Bullock's at 0.79 in California. The model narrowed 150 species to the two that were genuinely plausible, and the table answered a question the photograph was never carrying.

Two more factors sit alongside it. Time of day matters because an owl at two in the afternoon is a different proposition from an owl at two in the morning, and the app reads that from the sun's altitude rather than the clock, since a fixed 07:00 to 19:00 window is only right near an equinox. In Toronto, 19:00 is well after dark in December and broad daylight in June. Habitat matters because a duck deep in a forest is worth a second look, so the app estimates the ground you are on from OpenStreetMap land use. Each factor is optional and becomes a no-op when its input is missing.

Weather, and the Best Birding of the Year

The map view, which shows what is likely nearby right now rather than what you just recorded, adds a live weather read from MET Norway's Locationforecast. Weather moves birds on a timescale of hours, the timescale those pins should move on, and none of it reaches the occurrence counts, which know where a species turns up across a whole month but nothing about today.

Using it took two separate facts about each bird, because where a bird lives and how it feeds are hit by weather differently. Habitat covers exposure, so an open meadow takes the full wind while a wood stays sheltered. A second tag, the feeding guild, covers what the weather removes. Swallows are filed under meadow, right for where to look and no help in knowing that heavy rain takes their food, flying insects, out of the air. A freeze shows why one tag could not do both jobs, since the water habitat covers a mallard and a heron whose responses are opposite: ice concentrates waterfowl onto whatever open water is left, and takes the heron's shallows away entirely. Going from clear to heavy rain in Toronto in July drops the Barn Swallow nine places, off the map's pin list altogether, and lifts the Great Blue Heron by two.

Wind direction earns its place by being the one weather signal that makes birds more likely in miserable conditions. Most songbirds migrate at night and leave when the wind is behind them, a southerly in spring and a northerly in autumn, so the app checks the forecast wind against the bearing birds are travelling that season. Whether new birds arrived overnight swings on that, which is exactly what monthly counts average away: GBIF knows a warbler is around in May, not that this May morning followed a good flight. A tailwind alone suggests overnight arrivals, while a tailwind meeting rain is a fallout, where birds left on a good wind, hit weather, and were forced down en masse. It is the best birding of the year, and since every other term reads that sky as a reason to stay home, the fallout term is exempt from the rain penalty.

I set out to build a bird classifier and ended up building something closer to what a birder does without noticing, since nobody in a marsh identifies a bird purely from the bird. They use the bird, and where they are, and the month, and what the light and the weather are doing. The models are extremely good at the first, and the rest of the engineering went into giving every other source a question it was equipped to answer.

Songtrail is free on the App Store, and every piece of this runs on the phone in your hand. If you want to try it, it is at songtrail.grayforgelabs.com.