[webmaher]

I testing now with Plex and for now not bad at all...6710 series episodes imported in 27 minutes...over night i try with 180k episodes...
When you get new version just tell me 

[webmaher]

P.S. timed out again and again when i want import over 10.000 movies or 100.000 episodes....

[Montellese]

P.S. timed out again and again when i want import over 10.000 movies or 100.000 episodes....

Sorry to hear that and thanks for taking the time to test this again.

[webmaher]

i hope that you find a way to speed up this because i cant import complete movies and series... and as i have also  HBO addon is any chance to create also HBO importer...
Thanks for your work so far 

[LongMan]

@webmaher 

and as i have also  HBO addon is any chance to create also HBO importer...

Montellese could add MediaImport to the HBO addon but as he has stated a couple of times his time is limited and  it would be good to have addon developers familiar with the feature. Could you ask the developer of the HBO addon to add MediaImport support? The instructions are here.

Cheers,
LongMan

[Montellese]

@webmaher 

and as i have also  HBO addon is any chance to create also HBO importer...

Montellese could add MediaImport to the HBO addon but as he has stated a couple of times his time is limited and  it would be good to have addon developers familiar with the feature. Could you ask the developer of the HBO addon to add MediaImport support? The instructions are here.

Cheers,
LongMan

And I would need an HBO subscription which isn't even available in my country.

This whole thing will only work if other developers are willing to look into it. I can however totally understand if they are hesitant because this is not part of Kodi master / mainline...

[Montellese]

i hope that you find a way to speed up this because i cant import complete movies and series... and as i have also  HBO addon is any chance to create also HBO importer...
Thanks for your work so far 

So I've taken another stab at speeding things up. While I still haven't figured out how to improve python-plexapi to not perform unnecessary requests I started working on an approach where mediaimporter.plex uses background worker threads to retrieve the details of each media item in parallel. I've added a setting to specify how many background worker threads it should use since this highly depends on the machine / CPU that the add-on is running on. I've tried it with 10 worker threads (the default) and on my setup it speeds the process up from 165s to 55s to retrieve ~1k movies.

Furthermore I've started looking into better handling connection errors while retrieving items. It's hard for me to reproduce but I've added a retry logic which can be configured with the number of retries and the number of seconds it waits between retries.

@webmaher if you're interested you can give it a try by using the improvement/parallel_conversion branch of mediaimporter.plex. If you need a ZIP file you can get it directly from GitHub. Beware that I haven't spent much time testing it but I'm looking forward to your feedback and logs.

[LongMan]

@Supersilver,
Could provide some feedback on the latest improvements to mediaimpoter.plex mentioned above?

Cheers,
LongMan

[Supersilver]

@Supersilver,
Could provide some feedback on the latest improvements to mediaimpoter.plex mentioned above?

Cheers,
LongMan

I'll be sure to give it a go sounds like the improvements could help a lot, I'll get back to ye once I've tried it out.

Thanks Montellese for the work and thanks LongMan for bringing it to my attention.

[Montellese]

I may have good news for mediaimporter.plex: I managed to update python-plexapi from version 3.5.0 to 4.6.0 which offers a way to customize requests for item metadata. That way I was able to improve the performancy of retrieving 1k items by more than 50%. In combination with the experimental parallel metadata retrieval posted above the improvement reaches 75%.

Furthermore the new version allows access to some additional properties and to the TV episode intro markers avalable to Plex Pass users. I have a branch with a working "skip intro" feature.

Everything needs additional testing but I hope to release this ASAP.

[M4tt0]

That sounds great @Montellese! I'm not using Plex, so I cannot test it, but I keep testing the Emby functionality on my laptop and it's really phantastic!

I've realized that you have started to PR some of your media importing commits into Kodi master recently. Given the very advanced state your work is in now, are you planning to continue pushing your whole work into Master over the next couple of months or do you still want to keep it separate for longer?

[Montellese]

That sounds great @Montellese! I'm not using Plex, so I cannot test it, but I keep testing the Emby functionality on my laptop and it's really phantastic!

Thanks for the feedback. I just tested my latest performance improvements of mediaimporter.plex on a movie library with 9k+ movies. Last time I ran the import (back in March) it took 95 minutes. Now it took 6 minutes. That's not a bad speedup.

I've realized that you have started to PR some of your media importing commits into Kodi master recently. Given the very advanced state your work is in now, are you planning to continue pushing your whole work into Master over the next couple of months or do you still want to keep it separate for longer?

I'm still working on getting specific parts into mainline if they are useful for Kodi independent of media import. Right now there is one part which I would like to get into Kodi mainline but it needs some additional work to be suitable.

And I'm also still working on cleaning up stuff in my code which I did quick and dirty at the time. There's one major part in the code which will need a lot of attention and apart from that there's lots of undocumented code.

In the end it's still my dream to get media import into mainline but - apart from fixing the issues mentioned above - it will require the acceptance and support from the major part of the team. While I've received a lot of positive feedback on the idea and the approach only very few have actually given it a try.

[M4tt0]

Many thanks for your comprehensive response and explanations.

I fully understand what you are writing and sincerely appreciate your very dilligent approach. 

[...]

[...] While I've received a lot of positive feedback on the idea and the approach only very few have actually given it a try.

I agree again and that is where I think things should accelerate: I sincerely believe that the functionality you are providing should be a strategic priority for Team Kodi. I've posted the rationale a couple of months ago already: IMHO Kodi can only stay relevant if it continues to INTEGRATE all kind of popular media sources (stored media, TV streaming, streaming services, music, you name it) into one common ecosystem. That is what I believe users appreciate: One familiar and convenient platform to manage many things of relevance to the users. Kodi's add-on system was extremely visionary "at the time", but Kodi itself is currently being "demoted" to "just one app" within the ecosystems of the androids of the world. Today the choice on android is "Kodi App" vs. "Emby App" vs. "Netflix App" vs. "Music App", and no longer "Kodi App" and "Emby Add-on within Kodi" and "Netflix add-on within Kodi" and "Music section of Kodi".

YOU offer a solution to that, but - as you rightly write - too few users have tried it or even appreciate what you offer here: Your integrative approach should not just be used for Emby and Plex, but for EVERY media source provider seeking integration into Kodi. And the transition needs to happen WHILE you still have a significant user base. And I believe Kodi CAN prevail, exactly because you have no commercial interests and because you are not trying to abuse your ecosystem to change revenue streams. 

I understand and appreciate the voluntary nature of open source development and the Kodi project in particular. But bloody hell, I don't think you should need to handle all of this yourself! I'm no expert and my be wrong here, but intuitively, PRing more of your work quicker should lead to more testers and users and maybe also more developers helping and advancing the code.

Food for thought for you and Team Kodi from a long-time fan of the project, the community around it and the great work you have been doing here for so many years...

[Montellese]

That sounds great @Montellese! I'm not using Plex, so I cannot test it, but I keep testing the Emby functionality on my laptop and it's really phantastic!

Thanks for the feedback. I just tested my latest performance improvements of mediaimporter.plex on a movie library with 9k+ movies. Last time I ran the import (back in March) it took 95 minutes. Now it took 6 minutes. That's not a bad speedup.

Just ran an import from a PMS with 27k+ episodes which took ~15 minutes.

[Montellese]

Many thanks for your comprehensive response and explanations.

I fully understand what you are writing and sincerely appreciate your very dilligent approach. 

[...]

[...] While I've received a lot of positive feedback on the idea and the approach only very few have actually given it a try.

I agree again and that is where I think things should accelerate: I sincerely believe that the functionality you are providing should be a strategic priority for Team Kodi. I've posted the rationale a couple of months ago already: IMHO Kodi can only stay relevant if it continues to INTEGRATE all kind of popular media sources (stored media, TV streaming, streaming services, music, you name it) into one common ecosystem. That is what I believe users appreciate: One familiar and convenient platform to manage many things of relevance to the users. Kodi's add-on system was extremely visionary "at the time", but Kodi itself is currently being "demoted" to "just one app" within the ecosystems of the androids of the world. Today the choice on android is "Kodi App" vs. "Emby App" vs. "Netflix App" vs. "Music App", and no longer "Kodi App" and "Emby Add-on within Kodi" and "Netflix add-on within Kodi" and "Music section of Kodi".

YOU offer a solution to that, but - as you rightly write - too few users have tried it or even appreciate what you offer here: Your integrative approach should not just be used for Emby and Plex, but for EVERY media source provider seeking integration into Kodi. And the transition needs to happen WHILE you still have a significant user base. And I believe Kodi CAN prevail, exactly because you have no commercial interests and because you are not trying to abuse your ecosystem to change revenue streams. 

I understand and appreciate the voluntary nature of open source development and the Kodi project in particular. But bloody hell, I don't think you should need to handle all of this yourself! I'm no expert and my be wrong here, but intuitively, PRing more of your work quicker should lead to more testers and users and maybe also more developers helping and advancing the code.

Food for thought for you and Team Kodi from a long-time fan of the project, the community around it and the great work you have been doing here for so many years...

Thanks for your input and support.

The problem with PRing more of my work is that I'm close to the point where it's all or nothing. There may be a few things which can be left out from an initial PR but it's very little. There's basically nothing that can be left out without breaking the whole media import feature. The only thing I can think of right now is the integrated UPnP media importer which I haven't tested in a long time and which is probably not very relevante compared to the add-on / plugin integration.