Rendered at 21:56:29 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
Shershah1024 9 hours ago [-]
Hi all. I am Shahir. This was my experiment. First of all - Thanks to whoever posted this on HN. It came as a surprise.
Just to clarify - This started as just an experiment to see if we can use the fact that text and audio share a vector space in Embedding Gemma 2 to convert audio to text. Had to hack around a few things, but, it worked. Speed was just a side benefit. Was not aiming for it.
The accuracy is good, but not in the range of cloud models, but decent enough for everyday use, and this is just a couple of days work. The WER can improve a lot with more training on varied data. Will try that next.
I will opensource the whole thing soon. Might need a few days though. Also, releasing the TTS version for this soon.
Happy to answer any questions you have.
rahimnathwani 6 hours ago [-]
It was your post yesterday that made me aware of granite-speech-5.0-470m-turboctc (which I guess you used as a comparator because it's already very fast).
(Until now I've been using mlx-community/parakeet-tdt-0.6b-v3)
Shershah1024 4 hours ago [-]
Yes. Granite is really fast. IBM has actually been releasing some great models that are flying under the radar
xnx 8 hours ago [-]
Cool experiment. It will be amazing if this can match the WER of the best open weight models at better speed.
People aren't paying enough attention to how great EmbeddingGemma 2 and Gemma 4 are.
Shershah1024 4 hours ago [-]
I am experimenting with the reader with more varied audio. Plus, possibly an ultra fast post processing layer.
usern20260720 7 hours ago [-]
can this tool transcribe latin language texts? McWhisper allegedly does it but it outputs gibberish
cabarltos 8 hours ago [-]
Thanks for putting this out there!
e12e 6 hours ago [-]
[dead]
prodigycorp 8 hours ago [-]
can you turn this into a decision model
drakythe 8 hours ago [-]
What would a decision model add to and audio transcriber?
cheschire 7 hours ago [-]
Reducing cost for a large scale corporation call center. Press or say one.
Basic ML is probably even cheaper. But LLMs might have higher accuracy. “Uhh one I think” = 1
e12e 6 hours ago [-]
Theoretically - if the change was deeper than just restricting output - you could feed it thousands worth of audio tokens and get a structured response in milliseconds rather than minutes?
Say classify recorded interviews into viable/not viable candidate?
Shershah1024 4 hours ago [-]
It can already classify decently well, and I tried training it as a decision model, It worked well on domains I trained it for. It was beating Liquid's small decision model, but somehow, it never generalized. Out of the trained domains, it was just above chance.
sureglymop 7 hours ago [-]
Isn't EmbeddingGemma2 already a decision model? Am I misunderstanding your question?
whstl 7 hours ago [-]
For people interested in super quick models:
"One cool trick" I like for running those kinds of models very quickly on macOS is converting them to CoreML so they run in the Neural Engine.
I had amazing success with Spleeter, Demucs, MVSEP-MDX23-music-separation, DeepFilterNet, Whisper, Hunyuan3D for 3D mesh generation. Plus a lot of others for both work and leisure.
I also convert the code to Swift without dependencies, but that's a personal preference.
The runtime on some of them went sometimes from 60s to 10s. And since it's just Swift, I can just give them to non-technical co-workers and they don't have to setup a whole Python environment.
If you don't wanna do this by hand, this is something that existing LLM models are good at. Even Luna can do it without much fuss.
sbdaman 8 hours ago [-]
Maybe it's just a COVID time blending together effect but it still blows my mind how quickly we went from "automatic audio transcriptions are not even worth having on" to "an hour of audio transcribed in 13s locally."
Pre-pandemic it seemed like even the very best in class auto-transcription features were just unusably bad. I guess that was like 7 years years ago though.
gambiting 8 hours ago [-]
I'm just waiting for someone to tell the YouTube team. Hopefully 10 more years and someone over there will realize it's a thing.
7 hours ago [-]
f6v 8 hours ago [-]
[dead]
saaaaaam 8 hours ago [-]
I recently swapped to using Parakeet v3 on my old M1 Macbook Air, using MacWhisper. It was so fast I thought it had failed. As in, by the time I clicked "transcribe" and looked at my screen again a few seconds later the transcript had finished. Absolutely insanely fast.
1-6 3 hours ago [-]
People say old MacBook Air M1 but its architecture was so groundbreaking that it's still relevant in 2026. Apple definitely underplayed its release in 2020.
andai 25 minutes ago [-]
Nice. Would love to see a comparison with Parakeet and the recent Needle. Are there any benchmarks for transcription speed and accuracy?
llimllib 10 hours ago [-]
Apple's on-device transcription model is about 3x slower than their reported results on my more powerful computer, with a similar error rate in my test (which I consider pretty good results)
update: doesn't seem like they provide their model and I don't feel like trying to dig it out of their app bundle, so I'm not going to test this one
s3p 9 hours ago [-]
Is it even worth using apple's? Maybe they have improved it but when they first started using it to transcribe voice memos it was laughably bad (in my experience). I defaulted to whisper.cpp
solarkraft 5 hours ago [-]
I used it a lot in my AI client before I finally got around to hacking a Deepgram API server into Handy: It sucks enough to give me a headache worrying about its unreliability in every given transcription.
They supposedly improved it on later Apple Intelligence enabled devices, but I don’t have one.
llimllib 2 hours ago [-]
my tests are against the Apple Intelligence model, fwiw, not any earlier model
llimllib 8 hours ago [-]
apple's is 5x faster than `large-v3-turbo` with whisper.cpp and in my test had somewhat worse performance, so it's a tradeoff
ChrisMarshallNY 8 hours ago [-]
I suspect Apple's strengths are in multi-language support.
I use their on-device translation in my apps. Works pretty well. ChatGPT's context-aware translation is better, but it's not on-device.
bigtex 9 hours ago [-]
With MacWhisper you can select the Apple models, it took almost 4x as long to transcribe the same file versus this software and MacWhisper/Parakeet3.
llimllib 8 hours ago [-]
interesting! I only tested against Parakeet v2 because my agent suggested it would be faster for English transcription (v3 is multilingual but somewhat slower, it suggested)
On my bench, parakeet-tdt-0.6b-v2 is 2x slower
I wonder if you're using different Apple models when you go through that app? I don't know
bigtex 9 hours ago [-]
I tested a 33 minute local audio only file on a M4 Mac Mini with 24GB of Ram and it took 6.75 seconds to transcribe it using this software. Using MacWhisper/Parakeet3 it was about 7.5 secs using a stopwatch.
armcat 3 hours ago [-]
I applaud you on the experiment, this is a great showcase in how the latent space is shared between modalities and a great performance achieved.
You stated WER of 5% - against which benchmark?
Whistle [1] for example is a 16MB STT model but WER varies between 4-26% depending on the benchmark.
Is it open source? They mention something about the Apache License but I can't seem to find a link to GitHub or something.
cstever 10 hours ago [-]
I only see that EmbeddingGemma 2 is Apache 2.
"EmbeddingGemma 2 is made by Google DeepMind and released under Apache 2.0. This is an independent project, not affiliated with or endorsed by Google."
ghboo0927 9 hours ago [-]
The reference to Apache 2.0 on the page refers to Google's base model, not this app. Based on the page alone, the app is free, but the source code is not open-source.
This approach is interesting. Does this mean that I can limit the the words to the ones I actually use and build a tiny model that finds the closest match to only those words I use in 99% of my conversations?
drakirus 5 hours ago [-]
I'd be curious to see the WER on out of vocabulary words, seems like this cannot generalize to unseen word's given this step: 'The closest of ~229,000 English words wins, and the slices become a transcript'
Shershah1024 4 hours ago [-]
You can just embed those new words and they get added to vocabulary. It is fairly simple. One risk is - When a new word is very close to a known one, it tends to fall to it.
samuelbalogh 9 hours ago [-]
Fantastic! Any plans to support other languages?
Shershah1024 8 hours ago [-]
Yes. Planning to train a multilingual reader. Have not done a prototype yet.Will do one once I have the right dataset
samuelbalogh 8 hours ago [-]
Awesome work, thank you!
alberth 10 hours ago [-]
As someone who knows nothing on how this works, does this mean they have to pay the audio at 200x playback speed to transcribe this fast.
Or is there another approach?
treyd 10 hours ago [-]
It's not being played through a desktop audio stack, they're operating on the waveform decoded from the audio file. There's no constraints relative to the "natural speed" of the source, just like for video transcoding, it's limited by compute.
radlad 9 hours ago [-]
Yes but... Frame rate matters with video, which is arguably the video's speed.
treyd 6 hours ago [-]
You can flip through the pages of a book much faster than it'd take you to actually read them normally. But if you're putting the book in a scanner then you're still extracting information, faster than the "natural speed" of reading them.
sorou 10 hours ago [-]
during transcription, nothing is actually "playing" the file in the way a human would to listen to it. the audio is stored as a sequence of bytes representing the amplitude of the physical sound waves over time (the recording device measures, or "samples", the amount that its microphone is vibrating and records that amount, some tens of thousands of times per second) with some metadata and compression applied so that other software knows how to interpret the bytes correctly. this can be given to an audio program to convert the bytes back to physical sound waves using your speakers, but it's just bytes at the end of the day, so any software that can read bytes out of the file can do so at whatever speed it wants, regardless of how long the actual recording is.
so you have a file containing bytes representing the sound wave, and you have existing recordings (read: sound waves represented as sequences of bytes) of people speaking in different languages and contests, and you can feed all those existing recordings into a model along with corresponding transcriptions, and you can get a model out which can identify patterns in recordings of people speaking to then go backwards and see a pattern of sound waves and correctly associate it with a specific token, no need to play anything back. pauses, stutters, etc. are themselves patterns of sound waves that can be identified and worked around to identify just the meaningful parts of speech.
in principle, you could probably learn to do this too, by sitting down for hundreds of hours meticulously starting at waveforms, and this is, very roughly, what signal analysts do: look at waveforms and try to identify patterns. ai just leverages this principle to turn mundane speech into text.
cheschire 10 hours ago [-]
Probably significant parallelism. Split the audio into a lot of slightly overlapping portions to ensure the transitions between words are caught. I’m sure there’s some relatively simple ways to ensure you don’t double up on words.
drob518 10 hours ago [-]
Yea, I was wondering how they dealt with word boundaries and different word audio lengths generally (short and long). Your point about parallelism is interesting. Doing what they say they are doing, this should be ridiculously parallel, possibly with a reduce step at the end to remove redundant words if their slicing algorithm allows for that to happen. But yea, extremely parallel.
Shershah1024 4 hours ago [-]
To answer, words are not extracted in isolation. The model sees a context of about 20 words around it always.
satvikpendem 9 hours ago [-]
No more than an AI media generator creating the media in realtime.
9 hours ago [-]
p0w3n3d 5 hours ago [-]
The downloaded model is stored in
Library/Application\ Support/ai.haku.eg2demo
motoboi 7 hours ago [-]
A nice side effect is that it transcribes in English an audio in Portuguese. And it actually make sense.
But the overall transcription quality is very low.
Silagi 6 hours ago [-]
Is it low quality in the "It didn't match the meaning that was conveyed" sense, or in the "It didn't capture the exact words used" sense?
My reaction to seeing the project was that being an embedding model it'd be pretty good at conveying the meaning but not great as using the same terms, is that what you're seeing?
mahlerbox 5 hours ago [-]
Curious to hear the answer to this as well. Presumably it would be stronger on sentences with a high proportion of cognates/English terms with Latin roots? Does the English output over-represent words of Latin origin?
Shershah1024 4 hours ago [-]
The reason it generalizes to english now is because the reader is trained to identify English words, and it obviously understands Portuguese, so it just gave the text in English. Once I train it on multiligual vocabulary, it should be able to handle Portuguese
lukko 10 hours ago [-]
Interesting... is it overkill to use a multimodal model like this on just audio? Could the model size be decreased dramatically, or does the multimodality help the accuracy and speed in some way?
pjc50 7 hours ago [-]
The benefit is that you can take input in one "mode" and extract output in another; that is, the input is audio mode and the output is text mode. The intermediate representation is the "embedding".
sd9 9 hours ago [-]
Multimodality seems to just help with everything
coldtea 10 hours ago [-]
Correctly transcribing requires context too, no?
ranger_danger 9 hours ago [-]
I suppose it depends on the words in question... language knowledge and general world knowledge may not always both be necessary to transcribe accurately.
I think this seems to target English only I believe. Pretty good but thinking os using some of these techniques on our multi-lingual ASR models !
Shershah1024 8 hours ago [-]
Only English for now. I dont see any roadblocks for at least the major European Languages(Not sure about German though with their compound words)
martianvoid 10 hours ago [-]
Pretty cool application of an embedding model
chronogram 10 hours ago [-]
What about the quality of the transcription?
walrus01 10 hours ago [-]
They say its word error rate is about 5%, so not great. Interesting as a proof of concept. The model this is based on is REALLY tiny.
koolba 9 hours ago [-]
Too bad the errors for things like this are not truly independent.
Otherwise could run two passes to get down to .25% or three passes to .0125%. Or four passes to .000625%!
That’d be .02 seconds of error per hour!
robotswantdata 10 hours ago [-]
WER is not good enough for professional use, fine for hobby projects
cabarltos 6 hours ago [-]
You should add in a more comprehensive Audio Input selection - for example, selecting "Input 3" on my audio interface microphone
amelius 9 hours ago [-]
Sounds great if you could use it as a server. Otherwise, why would you want this?
etienne_l 9 hours ago [-]
[dead]
vladimirralev 10 hours ago [-]
Any info on the param size and quantization of each part and in total?
mikemzz1 10 hours ago [-]
interesting approach. But training ctc head on top of gemma's audio encoder will most likely produce better results and faster inference
Shershah1024 8 hours ago [-]
That's close to what this is: the reader is a CTC head on the audio encoder, but its output words are EmbeddingGemma's own text embeddings for ~229k words, instead of a separately learned vocabulary. I also tried a plain letter-level CTC head on the encoder. It did a bit better on clean audiobooks (7.0 vs 8.2 WER at the time) but much worse on harder audio like a lecture (16.5 vs 10.8), mostly from misspellings.
spiderfarmer 9 hours ago [-]
So Marco Arment can sell some more of his Mac Minis.
Aboutplants 9 hours ago [-]
First thing I thought of lol
guigoguedes 5 hours ago [-]
Does it work for portuguese?
jdw64 10 hours ago [-]
It would be nice if Korean also worked. I’d like to experiment with a Korean model too.
Imustaskforhelp 10 hours ago [-]
This has come at just the right time.
Does this work for Hindi or other non English languages though?
I wanted to recently transcribe Hindi + English (mixed) Audio and I used this model[0] and created a custom transcribe command which can give text over input of mp4 files
This took me roughly around an hour or more or around this time frame for a 90 minute transcription source. I would consider myself to be slightly bored during this phase.
I would really appreciate it if something like the project could be made native for a lot more languages but I think that the approach of embedding models and using them in such a way is pretty crazy. Kudos for it and I will try it
out!
I can think of many such use cases where Speech to text of even large audio files becomes near instant. As I think someone on hackernews once quoted whom I am very paraphrasing
An order of one or two magnitude of an already existing thing stops becoming a quantity upgrade but rather becomes a quality upgrade in it of itself as it provides access to workflows which previously wouldn't have been feasible.
Yeah I will have to try whisper as well. I didn't try it because I wanted to tinker more and also I had thought that whisper was an older model but I think that if it gets the job, then it gets the job done and I will be trying it out on the text that I mentioned, thanks for mentioning it as I will give it another go :-D
tinykerneldev 9 hours ago [-]
TrueScribe supports tranlating Hindi transcripts in both directions. Not free though.
bbor 7 hours ago [-]
Ok let's talk about the really important stuff here: that model name is atrociously long for such a game-changing model, and "embedding" is already an awkward phrase. I propose... Embedda?
Shershah1024 4 hours ago [-]
He he. Thanks for the suggestion. Just wanted to show that an embedding model can transcribe. Maybe we should call it emscribe :D
arbor-group 8 hours ago [-]
[flagged]
octowatch 10 hours ago [-]
[flagged]
mengyuxuan 10 hours ago [-]
[flagged]
bawana 9 hours ago [-]
I wonder if this approach can be used to transcribe the soounds made by dolphins and whales
Just to clarify - This started as just an experiment to see if we can use the fact that text and audio share a vector space in Embedding Gemma 2 to convert audio to text. Had to hack around a few things, but, it worked. Speed was just a side benefit. Was not aiming for it.
The accuracy is good, but not in the range of cloud models, but decent enough for everyday use, and this is just a couple of days work. The WER can improve a lot with more training on varied data. Will try that next.
I will opensource the whole thing soon. Might need a few days though. Also, releasing the TTS version for this soon.
Happy to answer any questions you have.
Thank you!
(Until now I've been using mlx-community/parakeet-tdt-0.6b-v3)People aren't paying enough attention to how great EmbeddingGemma 2 and Gemma 4 are.
Basic ML is probably even cheaper. But LLMs might have higher accuracy. “Uhh one I think” = 1
Say classify recorded interviews into viable/not viable candidate?
"One cool trick" I like for running those kinds of models very quickly on macOS is converting them to CoreML so they run in the Neural Engine.
I had amazing success with Spleeter, Demucs, MVSEP-MDX23-music-separation, DeepFilterNet, Whisper, Hunyuan3D for 3D mesh generation. Plus a lot of others for both work and leisure.
I also convert the code to Swift without dependencies, but that's a personal preference.
The runtime on some of them went sometimes from 60s to 10s. And since it's just Swift, I can just give them to non-technical co-workers and they don't have to setup a whole Python environment.
If you don't wanna do this by hand, this is something that existing LLM models are good at. Even Luna can do it without much fuss.
Pre-pandemic it seemed like even the very best in class auto-transcription features were just unusably bad. I guess that was like 7 years years ago though.
I'll test theirs too if I get around to it
Here's a sample CLI wrapping the apple model and providing a simple benchmark: https://github.com/llimllib/transcribe
update: doesn't seem like they provide their model and I don't feel like trying to dig it out of their app bundle, so I'm not going to test this one
They supposedly improved it on later Apple Intelligence enabled devices, but I don’t have one.
I use their on-device translation in my apps. Works pretty well. ChatGPT's context-aware translation is better, but it's not on-device.
On my bench, parakeet-tdt-0.6b-v2 is 2x slower
I wonder if you're using different Apple models when you go through that app? I don't know
You stated WER of 5% - against which benchmark?
Whistle [1] for example is a 16MB STT model but WER varies between 4-26% depending on the benchmark.
[1] https://cactuscompute.com/blog/whistle
"EmbeddingGemma 2 is made by Google DeepMind and released under Apache 2.0. This is an independent project, not affiliated with or endorsed by Google."
Or is there another approach?
so you have a file containing bytes representing the sound wave, and you have existing recordings (read: sound waves represented as sequences of bytes) of people speaking in different languages and contests, and you can feed all those existing recordings into a model along with corresponding transcriptions, and you can get a model out which can identify patterns in recordings of people speaking to then go backwards and see a pattern of sound waves and correctly associate it with a specific token, no need to play anything back. pauses, stutters, etc. are themselves patterns of sound waves that can be identified and worked around to identify just the meaningful parts of speech.
in principle, you could probably learn to do this too, by sitting down for hundreds of hours meticulously starting at waveforms, and this is, very roughly, what signal analysts do: look at waveforms and try to identify patterns. ai just leverages this principle to turn mundane speech into text.
But the overall transcription quality is very low.
My reaction to seeing the project was that being an embedding model it'd be pretty good at conveying the meaning but not great as using the same terms, is that what you're seeing?
Otherwise could run two passes to get down to .25% or three passes to .0125%. Or four passes to .000625%!
That’d be .02 seconds of error per hour!
Does this work for Hindi or other non English languages though?
I wanted to recently transcribe Hindi + English (mixed) Audio and I used this model[0] and created a custom transcribe command which can give text over input of mp4 files
This took me roughly around an hour or more or around this time frame for a 90 minute transcription source. I would consider myself to be slightly bored during this phase.
I would really appreciate it if something like the project could be made native for a lot more languages but I think that the approach of embedding models and using them in such a way is pretty crazy. Kudos for it and I will try it out!
I can think of many such use cases where Speech to text of even large audio files becomes near instant. As I think someone on hackernews once quoted whom I am very paraphrasing
An order of one or two magnitude of an already existing thing stops becoming a quantity upgrade but rather becomes a quality upgrade in it of itself as it provides access to workflows which previously wouldn't have been feasible.
[0]: https://huggingface.co/parismitaglobalsolutions/indicconform...
https://openai.com/index/whisper/
https://github.com/openai/whisper