MM 4 Chokes on Large Collections

[dav0043]

MM 3 was the only app I found that could manage my extremely large collection (close to 1/2 a million tracks). I had it tuned to be as clean as possible within the config options. No extra checks, no scanning, etc. I upgraded to MM4. It's great that video are now included BUT the app is now a complete dog with my collection. Just navigating around crashes the app at least 6 times a day. I've uninstalled, re-installed, configured and done everything I can to make it work to no avail. I've been hoping I'm not the only one with this problem but I have not seen a maintenance release in a while. Album art is especially buggy. Also, if I have anything in my Now Playing area the liklihood of the app dying goes way up. Is there any hope for MM4 to address this? MM3 worked fine (except for the constant disconnect/crash between the mini-player in the task bar and the main UI.)
The symptom for the crashes is no activity in task manager for the MM process. Occasionally it has activity and if i wait 15 minutes, it comes back from whatever excursion it was on.

[Lowlander]

Did you do the suggested File > Maintain Library with complete optimization?

[Rojer]

I think this problem might apply on my system with beta 4.05.90.

System
PC is XP3 with 3.5 gb Ram + 4 gb swap in Ram. i7/x58. Hard disk is data (not only audio) dedicated, hence 4kb clusters. 2TB. Healthy, contiguous file system.
Couldn't reproduce it but I once saw the whole 7 GB of virtual memory being used while doing some heavy work in MM. Don't know how MM handles memory but if there was a fix to deal with the 2 gb limit, it might have worked a bit too well... I think it was beta 4.05.89 at the time.

MM Database in ram, along temp files, thumbnails and such. FWIW, that virtual disk pushes 13 GB/s and has virtually instant access/seek time, compared with a hard disk, not just in benchmarks but in every real time application that uses it.
SQLiteSafety=0
Db got bumped up to 1gB in size, right after I copied MusicIp cpuid in custom field. Was a bad idea but makes for a great troubleshooting system.

I have a large db then: 230.000 entries. Despite this, everything "read related" is super fast : very good (better than MM3.2.5) tree-panel action with lots of "tree nodes-scripts" and collections. 1 second MM launch and ready time. Instant searches and auto playlists.

Writing is something else. First, disk writes seem to occur even when none should. At the least, MM seems to do some background work while doing any db edits. cpu idles at 12.5% (4 cores + HT).

I sometimes notice a progress bar for database update while editing large selections (1000+). I think it's too long since there is no disk access at all. Progress bar states it's updating db entries.

A fair example would be to change the season field from "03" to "3" for a whole TV show season [worth noting that column explorer displays two different seasons in that case], say 20 episodes. MM will just sit there, Gui unresponsive, doing apparently nothing for 1 minute, maybe more, displaying 12.5% cpu usage in task manager. Reorganizing on the same drive stalls MM as well and takes longer than renaming should. The slow performance applies to audio and video files, ony more noticeable with videos.
mask for music :

Code: Select all

t:\Data\Ma musique\Songs\<Extension>\<Genre>\<Artiste de l'album>\$If(<Année>,[<Année>] )<Album>$If(<Disque n°>,\Disque <Disque n°:2>)\<Piste n°:3>$If(<Artiste><><Artiste de l'album>, - <Artiste>) - <Titre>$If(<Extension><>mp3, [<BPM:3>bpm])

mask for video:

Code: Select all

T:\Data\mes Vidéos\Films\$If(<Séries>,<Séries>,<Titre> - <Réalisateur>$If(<Année>, [<Année>]))\$If(<Séries>,$If(<Année>,[<Année>] )<Réalisateur> - <Titre> - $First(<Acteur>,2),Film $If(<Disque n°>, - Part <Disque n°> )(Avec $First(<Acteur>,2)))

Not sure is this should slow down the renaming process, but this looks nothing like heavy processing for a i7 cpu.
TV shows have a straight mask (T:\Data\mes Vidéos\TV\<Séries>\Saison <Saison n°>\<Episode n°> - <Titre>) and reorganizing is just as painful.
Editing album arts freezes the Gui in the same manner; video still worse than audio.

The "stall time" appears to be proportional to selection size. The "Gui freezing" as well, meaning that MM will hang worse as the selection grows. Past a certain selection size, graphic corruption will occur and task manager will behave erratic. Windows explorer might hang as well on large edits, though it's not clear if patience would allow the system to recover. Other running programs do not seem to be hindered, as long as explorer and task manager allow access to them. After killing MM (when no more cpu cycles in use and gui still frozen), system sometimes needs a reboot, for MM will "fail to create image" at next launch. And "fail to initialize player" when nonskinned.

With db on ram and SQLiteSafety=0, you bet I'm doing 12 backups a day and then some... I'd like to help and troubleshoot this on my specific system. I probably need a couple of instructions on how to report this in a usable format.

[Edit]
I had to re-enable real time tagging. I then noticed that the behavior was no different than when disabled, except I could see the disk progress bar. Might it be that there is a small bug that would break "disable tagging ? In which case, behavior is almost normal, only a bit slow, like 300-400%, apart from the sluggish/frozen gui while tagging.

If that's all there is to it, might I suggest something being done about video files NOT being tagged as default. These need more editing steps than audio and MM does not appear to be ready to handle that task as transparently as with audio files. I don't suppose it should be expected to, video files being what they are.

I experimented with movies first, quite straight forward to tag but still more steps than audio. I then tried my luck with TV shows : this is where the fun begins, with different directors, writers, even actors per episode.
There is no way yet to tag TV shows in a few sweeps, unless someone comes up with some jmdb script. Which btw would be very nice. I wish I could write code and be the one ...

Bottom line :

• My take on the problem reported here, is that it might be related to tagging.
  Tagging the same video file over and over is just ridiculous, by design; it only makes it worse that it takes too much time and freezes gui.

While typing this edit, MM just tagged 1500 files, Flac and Mp3. That's still quite slow (I type slowly), considering the running system but that's quite acceptable.
What I might like to look into :
- frozen gui while tagging.
- disabling auto-tagging looks broken
- Tagging video should be a separate option, default should probably be off, maybe with a context item to freshen disk data.

I will install and troubleshoot the first beta release that include any of those fixes.
Maybe, if the problems reported here can be reproduced, a dedicated thread would be useful?

[Lowlander]

I sometimes notice a progress bar for database update while editing large selections (1000+). I think it's too long since there is no disk access at all. Progress bar states it's updating db entries.

This normally indicates it is updating the media files, it is not an indication of DB operations. Video files due to their size can take a while to tag.

And you should create the debug logs now so it can be fixed in a future release. Also note that the first thing to do is to run File > Maintain Library with complete optimization as that can result in a major improvement.

In my experience it seems that there are too many database locks causing hiccups when multiple actions are performed and it would seem that some of the SQL is inefficient asking for more data than is required for the current operation.

[Rojer]

Hi lowlander;
Confirmed. This is disk access indeed. This means that disabling tags is broken as suspected. I did not notice this in previous versions but I couldn't say when it started.
Db gets optimized twice a day. Wieth lots of improvements, since I am still doing large chunks of updates.
Today I decided to remove the MusicIp fingerprint from custom fields as it is useless anyway. This is what caused the db to inflate to 1GB. That's about 80.000 db entries and tags to update. 50 % done after 12 hours. A decent file manager will do 50 complete 2TB disk backups in the same amount of time. Mp3tag handled the transfer from fingerprint to custom tag in 5% of the time too. There definitely is something going on with file system access. I can confirm hardware is fine.

Video files due to their size can take a while to tag

Which is why systematic hard tagging doesn't make sense at all, since it is a multi steps process.

you should create the debug logs now

I have to do some reading on that: no clue what this means. I have the debug beta installed as standalone. Will this create the log just like that ? Wish I would have tagged my 80k files with it ...

[Lowlander]

This means that disabling tags is broken as suspected.

Do you mean to say the option to update tags on property edits is disabled under Tools > Options > Tags & Playlists?

have to do some reading on that

Step 4b: http://www.mediamonkey.com/forum/viewtopic.php?f=7&t=69

[Rojer]

Do you mean to say the option to update tags on property edits is disabled under Tools > Options > Tags & Playlists?

Yep, disabled or not doesn't seem to change a thing.

Running 1493 now. I re-enabled the option, since disabling does not help with the staling gui and I need audio files to have synched tags. Apart from stalling while tagging, performance is excellent for such a heavy collection.

MM 3.2 does a pretty fine job at handling files update in the background; MM4 obviously does not on my system. Is this a known issue or is it specific to my system ? In which case I should build a log while tagging and send it.

Do you think something could be done with the systematic video files updates ? I don't see how this could ever be useful, unless a way is found to drastically speed up the process...

[Lowlander]

For me it has been improving over the versions and haven't had any troubles with tagging. Although I believe I read somewhere that for certain video types the tag changes required full file rewrite because tags were at the beginning. This may be your issue.

You can always send a debug log (step 4b) to support capturing these slow downs: http://www.mediamonkey.com/forum/viewtopic.php?f=7&t=69

[dav0043]

I've Optimized, re-optimized, re-created libraries, re-started, re-booted, etc. It just goes off into never-never land when it feels it needs to scan through something (the library db? the windows file structure?). I too have an i7 8GB 8 virtual processors so it's not the machine. If open the media tree, wait for 10 minutes or so, and don't make any changes and don't close or open anything on the media tree, it seems OK. I must be in the minority here with this huge collection but this is why I switched to MediaMonkey. If/when I have another choice I will run screaming to it. I'm a tech-comfortable person who will do whatever I can to avoid bloatware, but here we are with the latest versions being bloatware. Trying to do way too much. Maybe the people with small libraries are happy and I'm the odd guy out but it's a drag.
Also, with video tagging, I ran some tests by labeling videos and syncing the tags manually. It happily does it. When I check the files, no changes applied (no genre, actors). Tried for Video, Music Video and TV Shows. Same result. Checked the options on Tagging to ensure they were on. Doesn't work. Just to test something I clicked on the 'refresh tags from file' option and tried another sync. No change. Quickly turned off that option and continued on. Went back to my video and the tags were blown away. I know it's not as fun to code fixes for basic tagging and library navigation but it's kind of essential.
I'm really a big fan and can imagine you guys do this in your spare time, so thanks. It is just frustrating to watch a product with such potential go, in my opinion, the wrong way.

[MMuser2011]

One point I have found: If you display some kind of covers, the start process takes several minutes (just to show about 20-50 first entries in your choosen sort order). If you close MM4 on a node like "Files to edit" and then re-start MM4, it needs only about 30-60 seconds until all entries are read from the MM.DB (this is still much too long compared with other databases - not music related), but I don't see a way to improve this.

I would like to see some speed improvements too.
(Btw: I can confirm, that Mp3tag read and write the identical amount of files muuuuuch faster.)