AI · 14 Nov 2026 · 8 min read

What we learned analysing 24,000 sales calls with AI that never leaves our office

Our callers phone hundreds of businesses, students and parents every day. Until this year, reviewing those calls worked the way it does in most teams. Someone downloaded recordings by hand from the telephony dashboard, listened to whatever they had time for, and passed feedback along later.

That approach has three problems. It covers a tiny sample. The feedback arrives after the caller has already made the same mistake a hundred more times. And the information that matters most, like who was interested, what they objected to and when we promised to call back, ends up in people’s heads instead of a system.

So we built one. Since July 2026 it has processed more than 24,000 real calls, over 250 hours of audio, in Hindi, Hinglish, Punjabi and English. Here’s what we built, what went wrong, and what we learned.

The goal: every call heard, only some calls listened to

We didn’t want to replace our reviewers. We wanted to point them at the right calls.

For every recorded call, the system should produce a short English summary, an outcome, a lead temperature, the objections raised, any follow-up date that was actually agreed, and a caller-quality score with a coaching note. Then simple rules decide which calls a person should check:

• hot leads
• unhappy customers
• low caller-quality scores
• calls the AI isn’t sure about
• a small random sample, so the AI is never left unchecked on ordinary calls

Today about one call in four gets flagged. Everything else is summarised, scored and counted without anyone needing to listen.

Rule one: the data never leaves the building

Call recordings are some of the most sensitive data a business holds: customer phone numbers, family details, what people can afford. Sending them to a cloud AI service was never an option for us.

Everything runs on a single Mac Studio in our office:

• Speech-to-text with whisper.cpp, an open-source engine for OpenAI’s Whisper speech models, running locally on the Mac’s GPU.
• Analysis with an open-weight language model served on the same machine.
• The review app on the office network only, with role-based logins and a log of every transcript view and audio play.

No audio or transcript goes to an outside AI provider. The database is backed up nightly to a separate network storage server, and retention periods are configurable to support the DPDP Act.

The pleasant surprise was speed. A typical call took about six seconds to transcribe and five seconds to analyse even in our early laptop tests, and the Mac Studio is faster. A full day’s calls are done long before the next morning’s stand-up.

Surprise #1: most sales calls are tiny

Before building anything, we measured a pilot set of about 2,600 calls. The median call lasted 25 seconds. More than half were between 10 and 30 seconds, and only 1 in 100 went past two minutes.

That changed how we thought about the problem. Most outbound calls aren’t sales conversations. They’re openings: “Who is this?”, “Not now”, “Call me in the evening”, “Wrong number”. The AI has to be good at short, messy exchanges, not just long, polished ones.

It also meant that a large share of calls shouldn’t be analysed at all. Calls with almost no speech are set aside automatically as “too short”, which saves time and keeps them out of the statistics.

Surprise #2: the AI said “nobody talked”. Mostly, they had.

Our first full run labelled 74% of analysed calls as “no conversation”. That couldn’t be right, so we checked. In those calls the customer’s side had about as many words as in calls labelled “not interested” or “callback”.

There were two causes:

1. Our definitions were loose. To the AI, a customer saying “nahi” and hanging up looked like “nobody really engaged”. To us, that’s a clear “not interested”.
2. The transcripts were garbled. The speech model we started with struggled with phone-quality Hinglish, and the AI couldn’t follow what it couldn’t read.

We fixed both:

• Tight, ordered definitions. The AI now checks outcomes in a fixed order, and “no conversation” is allowed only when the customer never responded at all. Real no-conversation calls are split into reasons: IVR or switched off, voicemail, busy, silent hang-up or unclear audio.
• Measured facts in the prompt. We measure how long each side spoke and give that to the AI, so it can’t claim nobody talked when the customer spoke for 15 seconds.
• Evidence, not just answers. The AI must quote the transcript lines behind its outcome and follow-up date. Reviewers see those lines highlighted and can check them in seconds.

Across the full set of calls, “no conversation” is now about 26%. That’s a different, larger set than the pilot, so it isn’t a lab measurement, but the change is unmistakable. Most calls now get an outcome a manager can act on.

Surprise #3: the right speech model matters more than the “smartest” one

We compared four speech-to-text models on the same 40 real calls: two standard Whisper models and two community models fine-tuned for Indian speech.

The winner was Hinglish-Apex, a Whisper model fine-tuned on noisy, Indian-accented audio that writes Hinglish in Roman script. It was:

• the fastest: 3.9 seconds per call, against 5.9 seconds for our starting model
• the cleanest: no repetition loops, where the model gets stuck repeating a phrase
• the easiest for the AI to read: the AI rated only 5% of its transcripts low-confidence, against 22% for our starting model

We also learned how much an outcome depends on the transcript. Only 11 of the 40 calls got the same outcome from all four speech models. If the words are wrong, the analysis is wrong, however good the language model is.

That’s also why every campaign now has a vocabulary list. Speech models reliably mishear brand names, course names and place names, sometimes into words that mean something else entirely. In one early test the model turned the Hindi word for “Monday” into something the AI read as “Sunday”, and a follow-up date was off by a day. Correcting known mishearings before analysis fixes a whole class of these errors.

Surprise #4: one set of instructions doesn’t fit every campaign

Our team calls for more than one kind of campaign. “Interested” on a website sales call means asking about pricing or a meeting. On an admissions call it means asking about courses, fees, hostels or scholarships. And “already has a provider” means a different web agency in one campaign and a different college in the other.

So the instructions are versioned and chosen per campaign, and sometimes per caller. Each result records which model and which instruction version produced it, so we can always compare old and new on the same calls before switching.

Keeping humans in charge

The AI drafts. People decide.

On the review screen, the reviewer sees the summary, the transcript synced with the audio (click any line to hear that moment) and the AI’s verdict. If it’s right, one click confirms it. If not, they change only the wrong field. Corrected values replace the AI’s everywhere: in totals, the dashboard and the daily digest.

Reviewers also rate the transcript and summary, and write down what the AI got wrong. More than 1,000 calls have been reviewed so far, and each one is a labelled example. We use them to:

• measure how often each AI model agrees with our reviewers on our own calls, rather than trusting benchmark scores from elsewhere
• find the disagreements between models, which are the most useful calls to review
• refine the instructions, and only ship a change after checking it against reviewed calls

A few safeguards we consider non-negotiable:

• No invented follow-up dates. If no day was agreed on the call, the field stays blank.
• The AI flags its own doubt. Low-confidence calls always go to a human.
• Old results are never thrown away. Reprocessing a call with a new model keeps the earlier result for comparison.

What changed for the team

• Callers see a summary and a coaching note on every call, plus the follow-ups they promised.
• Managers start the day with a digest: calls per caller, outcomes, top objections and follow-ups due. The dashboard shows the funnel from recorded calls to conversations to interested leads, per caller and per day, and every number links to the calls behind it.
• Reviewers work through a short queue of calls worth hearing instead of a pile of recordings.
• Everyone can see which objections come up most, which is the start of better scripts and better training.

What’s next

• Real-time intake. Calls will be picked up automatically minutes after they end, through our telephony provider’s call-record push, instead of from daily exports.
• CRM write-back. The summary, outcome and follow-up date will land on the right lead record, matched by phone number.
• Digest delivery by email or WhatsApp.
• Learning from corrections. As reviewed calls add up, we’ll use the closest corrected examples to guide the AI. Eventually we’ll fine-tune a smaller, faster model on our own labelled calls.

Want this for your team?

We built this for our own callers first. Now we offer it to other sales and telecalling teams as NetGen CallSense: for colleges and coaching institutes, hotels and travel companies, real estate, healthcare, and banking and insurance teams across Himachal, Punjab and the Chandigarh tricity. We can host it for you on our private servers, or help you run it on your own hardware so your calls never leave your premises.

If your team makes hundreds of calls a day in Hindi, Hinglish, Punjabi or English and hears only a handful, we’d like to show you what the rest are saying. Start with a free pilot on a week of your past recordings.

Read about NetGen CallSense →