Weekly AI newsletter
First, let's release.
[ Previous: simscan+DeepSpam :: Rattlegram in C :: Neuromancer SDL :: Telegram-CLI and iPhone ]
I've finished the port of the good old xTTSv2 architecture (2023) into... well at first it was supposed to be CrispASR, but Qwen 3.6-27B couldn't manage it (and I don't know shit about shit), so I switched over to audio.cpp, and that of course didn't change much at all, but still, this is where GLM-5.3-Flash-IQ6 finished the job.
"Oh but it's sooo 2023", and it really is, but the problem here is that, first, there are ZERO other TTS models with good hungarian language support. OmniVoice makes an attempt, but it was trained to be completely retarded, and any short spoken expression has a 50% chance to come out as a yell or anything else really... And in english, the delivery is so utterly monotonous that it's almost physically painful to listen to. I guess it sounds good enough for the NPCs out there?..
original speaker voice (Amber, the ridiculously hot chink babe from Prof. Jiang)
OmniVoice sample (AUDITORY RAPE WARNING)
xTTSv2 (soothe your ears after the shock)
Now, the issue with audio.cpp is that the repo's name you'll never remember, but also that I have no idea how to run it. Basically, the process is taking an input TXT, and producing an output WAV. Simple enough, eh? Unfortunately, the dev managed to make this process as obtrusive, opaque, convoluted, unsolvable as possible, with the user having to maintain not one but two JSON config files, as if one of these incredibly user-hostile formats wouldn't be insulting enough to be used in such a role. The CLI interface requires 6 million options to be specified, because it's too fucking stupid to figure out ANYTHING by itself. So here's a cheat sheet:
Place the GGUF files in the subdir in the command line. If you don't specify the fucking "spec" JSON, audiocpp will shit itself when it finds the vocoder's GGUF file (which is itself mandatory, so crashing on this is quite unforgivable).
audiocpp_cli --task tts --family xtts_v2 --model-spec-override d:/llm/xttsv2/xtts_v2.json --model d:/llm/xttsv2/q8_0 --backend cuda --threads 16 --batch-text-file %1 --batch-merge-audio concat --voice-ref d:/llm/xttsv2/amber04-30s-22k.wav --out a.wav --batch-merge-silence-ms 500
audio.cpp's default operation when processing text in batches is merging them with 120msec between segments, which is aneurysm-inducing, so I added the --batch-merge-silence-ms option (server endpoint also accepts it in the request JSON), which really should be a default functionality with a value of 500.
Another issue is that the dev took the implementation and refactored it with Claudia, and now it's both unable to use the Q4_K-quantized model (for some reason?..), AND entirely dependent on the "upstream" GGML lib, which has an absolute horseshit performance on CPU builds, but I guess it's better for CUDA (I never tested mine on that target, I'm too destitute to own an nVidia). In short:
- for CUDA/other GPU, use my tree with at least revision 989a37e9, which has the batching silence option. This is the Claude-refactored tree, so only Q8_0 works and I'm not fixing it because I don't use it.
- for CPU, use my "ik_llama" branch, and compile with
-DAUDIOCPP_GGML_SOURCE_DIR:PATH=/wherever/your/ik_llama.cpp/ggml. It's extremely fast compared to niggerganov's slow GGML. This is entirely incompatible with the 'main' tree, and will never be merged, but who cares. On a 48-core Xeon Silver it's almost realtime, 1.1 RTF. - for the HTTP endpoint server, here's a quick sample config JSON.
In other news, I've finally evaluated some cloud providers, having never (!) used any of them before in a paid manner (ever since experiencing beta.character.ai's fuckery), I'm extremely averse to anything not local. Anyway.
Turns out it's such a fucking face-melting scam even I was appalled... The issue is with "caching". In this context it means saving your request's KV cache data (which is your prompt itself processed ["prefilled"] into LLM-digestible format, a parallelized, fast, but still a computationally-intensive operation) unloaded from VRAM either into system RAM or simply storage. Which is a basic I/O op of few gigabytes, ancient technology, right? WELL TURNS OUT THEY TAX THIS HARD AS FUCK, and at a meager 256k-ish token prompt, this essentially free operation takes up 90% of your costs, landing brutal fees PER TURN, PER TOKEN! OpenAI, Anthropic, and even the chinks do it. EXCEPT DeepSeek, which is almost completely free of this charge.
Guess what I use now, when not local Qwen/GLM.
Lastly, my PicoLM LLM inference software is nearing release. This will be a separate entry. Because it warrants another lust-inducing image.
[ Listen to RADIO AMICIA for a live technological showcase of all this, accompanied by 24/7 AceStep music ]