+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Discussions (https://forum.kodi.tv/forumdisplay.php?fid=222)
+--- Forum: Feature Requests (https://forum.kodi.tv/forumdisplay.php?fid=9)
+--- Thread: Beta Automatic A.I Translation of Subtiles (/showthread.php?tid=376459)
Automatic A.I Translation of Subtiles - NeOHM - 2024-02-28
Hi,
Often we find quality contents which have integrated subtitles or subtitles downloaded using addons but their subtitles are only available in english or another language which is not suitable for some of us. With the raise of A.I it is now possible to translate efficiently subtitles in languages where they are not available.
It could be an interesting feature to add kodi ability for automatic translation of the played subtitles whatever their source (included in the video, local to video's folder, or downloaded by a subtitle addon) using one of the many A.I services available now. Like ChatGPT, Gemini, Bing, Claude....or at least Google Translate through a simple API call to these services.
A Very useful feature imo
RE: Automatic A.I Translation of Subtiles - izprtxqkft - 2024-02-28
would be simpler to just buy a movie that is actually in the right language and copy the subs off the disc when backing up
you have to consider most subtitles are graphic based so would have to use OCR first just to convert them to text based and then you could use one of your auto-translations
but OCR already has problems recognizing characters and computer translations are often grammatically incorrect
RE: Automatic A.I Translation of Subtiles - NeOHM - 2024-02-28
I am not talking about subtitles graphically inserted in videos, of course these ones cannot be translated without OCR. I am talking about the embedded subtitles in an mkv track, local srt files or even subtitles downloaded automatically by subtitle addons. These are simple txt subtitles that can be pushed to a translation service or an A.I through a.simple call. Of course the translated result need to keep original timecodes of source subtitle.
RE: Automatic A.I Translation of Subtiles - izprtxqkft - 2024-02-28
(2024-02-28, 02:37)NeOHM Wrote: I am not talking about subtitles graphically inserted in videos, of course these ones cannot be translated without OCR.neither am i, PGS subtitles are graphical, ASS subtitles are graphical
(2024-02-28, 02:37)NeOHM Wrote: I am talking about the embedded subtitles in an mkv trackmkv is just a container, those subs are PGS, ASS or SRT
(2024-02-28, 02:37)NeOHM Wrote: local srt files or even subtitles downloaded automatically by subtitle addons.
sure, or just paste the plain text subs into https://translate.google.com 1 time instead of every time a video is played
what you're talking about only really benefits downloaded subs, retail discs only have PGS or DVDSub which are graphics, basically an overlay jpg in simple terms
RE: Automatic A.I Translation of Subtiles - MatteN - 2024-02-28
I still haven't seen any AI make a decent translation, they are pretty much garbage all of them.
RE: Automatic A.I Translation of Subtiles - jjd-uk - 2024-02-28
A non-starter as you have to pay to use the API's.
RE: Automatic A.I Translation of Subtiles - NeOHM - 2024-02-29
I was talking mostly about TXT SRT files downloaded by addons like OpenSubtitles, A4ksubtitles,.... when watching my ripped DVDs and Blueray movies and TV Shows which don't have French subs ;-)
Of course other ASS graphical Subs couldn't be translated. I don't use the embedded subtitles.
RE: Automatic A.I Translation of Subtiles - NeOHM - 2024-02-29
I have a ChatGPT Plus account, this is not an issue for me.
Also the feature could also support local A.I translation through a localhsot API call, as some A.I projects provide trained A.I software to install locally.
RE: Automatic A.I Translation of Subtiles - jjd-uk - 2024-02-29
Quote:Please note that the ChatGPT API is not included in the ChatGPT Plus subscriptionfrom https://help.openai.com/en/articles/7039783-how-can-i-access-the-chatgpt-api and as far as I can tell it's the developer who has to pay.
RE: Automatic A.I Translation of Subtiles - grandpayum - 2024-03-26
(2024-02-28, 16:04)jjd-uk Wrote: A non-starter as you have to pay to use the API's.
you can host the model locally and generate and translate subtitles without needing to pay but it's like 7-8GB and the translations are not good
https://github.com/McCloudS/subgen for a project that does exactly this - a docker image with the openAI whisper model and python scripts to generate subtitles based on the media sent to it. Does a pretty great job with transcribing media actually; I have it on my server for exactly that purpose and it has replaced pulling subs from opensubtitles and the like. but every time i've tried it for translation of foreign media it results in a jilted difficult to understand mess because it doesn't do any kind of localization whatsoever. 0 understanding of colloquial language, slang, etc.
RE: Automatic A.I Translation of Subtiles - subdog - 2024-10-28
(2024-02-29, 17:08)NeOHM Wrote: I have a ChatGPT Plus account, this is not an issue for me.I originally thought so too, but recently I have seen some papers and cases about reflective translation. It seems that the quality of artificial intelligence translation is now only related to cost. When you are willing to use some models that cost a little money, along with prompt words, the improvement in results is very noticeable.
Also the feature could also support local A.I translation through a localhsot API call, as some A.I projects provide trained A.I software to install locally.
RE: Automatic A.I Translation of Subtiles - mYnDstrEAm - 2024-12-02
Good idea, Google Translate Batch works well and this would be a very useful addition for all the cases where you have subtitles in some language but not in the language you look for and it's not available in the subtitle addons you have installed. Just select "auto subtitles" for the selected language and use subtitles that are mostly correct. It could also be used by those subtitle addons so that subtitles are create more quickly and with less effort.
People who commented in this thread seem to base this on results years ago. MatteN, that is just false by now. ChatGPT should not be used as it's inaccurate and is not open source. jjd-uk and subdog no, one doesn't have to pay for that. The software by grandpayum also looks interesting and maybe this and the linked could be hosted by one of the subtitles addons but it's mostly about translation anyway, transcription is not needed if you already have one subtitle file.
You could even let an AI tool create new subtitles for some video but that could take quite long and the resulting file should probably be shared so other people can then just use this subtitle instead of waiting long themselves. Maybe it would be best to raise this at one of the top subtitle addons.
RE: Automatic A.I Translation of Subtiles - izprtxqkft - 2024-12-02
(2024-12-02, 20:06)mYnDstrEAm Wrote: Maybe it would be best to raise this at one of the top subtitle addons.
https://forum.kodi.tv/showthread.php?tid=357986
RE: Automatic A.I Translation of Subtiles - stgiga - 2024-12-10
Apparently, font-level machine translation is possible on anything compiled with modern HarfBuzz with a special flag. It's done via a project called Translate.ttf, which uses HarfBuzz's Wasm Shaper. Translate.ttf works fine in VLC once you use the right build of HarfBuzz. Of course, I'd be remiss if I didn't take the Wasm table of Translate.ttf (doing anything besides English-to-German requires building it from source) and shove it into UnifontEX, which I did, using ftxdumperfuser on an old Mac of mine, which funnily enough ran Kodi earlier in its life. I've been a Kodi user since 2013.
Honestly, the fact that one has to choose between Arial or that other font usable for subtitles completely leaves out anything in any CJKV scripts, or really anything that isn't ASCII, and so truthfully I feel like UnifontEX would make a great fallback font in case a character is in neither font, or usable directly for those who like retro captions.
UnifontEX supports Plane0+1 (also some notable characters from Planes 2 and 3, and the entirety of Plane 14) in one TrueType font (to honor the OpenType tables present that I added, I strongly suggest referencing it as "UnifontExMono-VF.otf", because it DOES have OpenType tables but its outlines are conventional TrueType format, not CFF), something that regular Unifont doesn't do.
There's actually a lot of cool things Plane merging enables, because of how certain notable sections of Unicode were allocated. Most of the feats on the demo page aren't possible in upstream Unifont.
Also, UnifontEX is based on Unifont-JP 15.0.06 and Unifont 11.0.01 Upper. That's very good Unicode support.
As for why the JP version, well, it had certain Han characters not in the base version, including special ones like Biang and Taito. Also the Katakana is more readable.
Ironically, there's quite a lot of Chinese fans of the project (users of BOTH TC and SC). Meanwhile if you use non-JP Unifont (Simplified Chinese) for representing Kanji, Japanese users are quite vocal in their distaste for that.
So in theory, it's a one-and-done.
Now the question becomes whether or not Kodi is even able to bundle Arial. UnifontEX inherits Unifont's GPL2.
If Kodi *can't* bundle Arial, there's definitely a vacuum.
Now then, I should mention that the machine translation at font-level with the lightest model is much too large for inclusion in Kodi. Like, the resulting Wasm table is in the tens of megabytes, and unlike the rest of TrueType (even stuff like the elusive VDMX table), it isn't compressible, likely being already compressed.
The Wasm table doesn't have inherent compression though, and the reason it has to be manually compiled is that the feature requires some responsibility.
But yes, font-level machine translation IS possible, but doing so would effectively hike Kodi's file size due to the incompressible table. Meanwhile UnifontEX as-is, no Wasm table, can go down to 3MiB in DEFLATE, or ~2MiB in LZMA2. So at the very least, some good can come out of this idea.
I know that the problem with using API methods for neural translation is that they can be flaky, a problem affecting other types of addons on RPi, well, at least when not running it on Raspbian without LXDE removed. API keys can be a big nightmare to deal with.
Especially without a keyboard.
I can see the appeal in wanting to mitigate such approaches via local code. That's how I run my AI experiments. The problem then becomes available resources, and not one type.
Ultimately, this cool stuff (including AI voice recognition which I haven't even covered) is cool, but it needs to be done carefully and properly.
Unicode support enhancement is still needed though, and thankfully easier.
Of course HarfBuzz went beyond 65535 glyphs and I do plan to target that in UnifontEX2 (which will also do other cool stuff with the Wasm feature), but no tools exist to generate such fonts yet (the idea is to get parity with Unifont 16+ but still one TrueType and the PUA stuff that they have), and only HarfBuzz supports it.
Anything else only would see UnifontEX's glyphs, not the new ones unless using 2022 HarfBuzz or newer.
Does Kodi support HarfBuzz (note: it's not mandatory for regular UnifontEX) at all? If so, how new is it?
My apologies if this post isn't the best.
RE: Automatic A.I Translation of Subtiles - stgiga - 2024-12-10
The forum isn't letting me edit, and there's some more details I need to cover: translate.ttf IS AI machine translation via LLaMa, put into the Wasm (for WebAssembly) table. The idea of an LLM in a font via this route was actually "llama.ttf", which generates short stories with the right input. Translate.ttf will output to console. The speed on WSLg is *just* fast enough to match the video in VLC (I use a Raptor Lake i9 with RTX4060.)
Making these fonts display a lot of text in a short time can be laggy or worse. Meaning font-level machine translation should be opt-in.
Now, the Wasm table can live in fonts of either TrueType outlines or CFF outlines, but the HarfBuzz extensions to glyph count aren't for CFF. They did however make TrueType able to use the same cubic outlines as CFF (apart from em size). UnifontEX doesn't use CFF. I had gone to bed before I could fully make my point.
Also the "Mono" in the font name is one of multiple strategies for making the font show up in IDEs and Terminals (it's an excellent debugger font). As for fixed-with fonts being used in captions, this was and is done on many set-top boxes, including external closed-caption decoders. Ceefax and crew were also not proportional fonts. So there IS precedent for using a monospaced font in captions.
Also from experience, American captions include symbols like music notes, and Japanese captions via ARIB actually include quite a lot of special characters, and UnifontEX handles them well.
UnifontEX also has MANY other types of symbols, plus many languages. FAR more Unicode than Arial Unicodde MS. It's the lions' share of Unicode, making it great for this type of use case, among others. It's also MUCH smaller than Noto. If there's anything else you want to know, let me know.
Hopefully my college exams go well.