MM 3 Alpha 2 - Large Library Performance Related Observation

[Randall Grogan]

Rusty - Keep the feedback coming! It really helps us make the product better. There's so much of it coming that we're not able to keep up/respond to all of it, but we are listening--most of the fixes fixes are based on bug reports that you've sent. btw, I'd be very interested in hearing performance-related feedback from users with large collections (25k+ tracks).

I am in the process of loading up 106k tracks. A mix of M4A and MP3, comprising 770 gb data.

FYI... It was very exciting to see MM3 read ACC/M4A tags !

My first performance related observation is that this load is going to take a "unit of measure" days. CPU is running at 70-80%, with tracks loading at about 20 tracks/minute. I do have it "Analyzing Volume" and looking for duplicate tracks, which slows the process.

Let me know if there are any specific events you would like tested. I could compare similar functions between MM2 & MM3?

I previously had 22k tracks loaded in MM3 Alpha 1. The overall performance with that many tracks was acceptable, and in general seemed faster than MM2. We will see what happens when I load the remaining 84k.

[rovingcowboy]

in winXp right click on taskbar click on task manager and go to tab with all running programs listed.

find the media monkey exe for the mm3 version. turn the 2.5.5 off if your running it to play music while the other one does it stuff? if that is even possible to do?

but right click on that media monkey exe and go to the option to set priority and set it in high if your doing other stuff on the computer.

if your computer is just sitting there letting media mo nkey run then set it on realtime. do the same for your mouse driver put it on real time or your mouse will be hard to control when it is not as fast or faster then monkey.

now close the task manager.

go back to media monkey. right click on the status bars you have showing each one at a time and set that priority to highest.

now your songs will be done a lot faster i went through 2000 in less then 20 minutes just adding them in to the library and anaylizing the track volumes.

after your done when you close monkey the task manager setting will go back to default for the next time you start monkey.

[Teknojnky]

Setting app priorties higher than normal is generally BAD. That prevents other apps and system processes from running. If you do that a lot no wonder you have such problems with your windows.

[rovingcowboy]

that is not the reason for the errors in media monkey you might want to think again about saying that.

the setting of those prioitys that i mentioned is fine since they all go back to normal when you close media monkey. and or media monkey finishes the scanning or anaylizing tasks. all but the mouse that is. but the mouse pointer software has nothing to do with media monkey's software.

and when you speed up the tasks to finish them faster there is no harm in it since it is just that one task. it only slows down the use of everything else which is why i said to speed up the mouse also. and on a reboot of the computer the mouse is set back to normal or you can reset it yourself.

you think the microsoft devlopers of the os sit there in slow mode to test it? who do you think put that there. it was not me.

[Randall Grogan]

24k loaded, 82k to go. This first 20k should be the fasted, since I am rescanning a portion of the music libary that was already added to to MM3 Alpha 1.

RovingCowboy, CPU load is running at 85%, so changing priority will not help significantly. No worries, I have 3 desktops. My music library resides on a NAS device.

[Randall Grogan]

RovingCowboy, I should also note that MM is the only application running on the PC right now.

I will be curious to see if the application slows down as we approach the 100k mark.

[rovingcowboy]

oh thats good i'm glad its going fairly fast for you.

i have a small cpu is why i do the speeding up.

the speeding up of the task inside of media monkey by
clicking on the status bars is to adjust the prioity setting
inside of mediamonkeys code. so that will speed up even
if your cpu is running almost at full usage.

[jiri]

We are certainly interested in such large libraries testing results, so keep them comming. Although we make some internal tests, nothing is as good as testing on real data.

Jiri

[Teknojnky]

I have 100k tracks, 650ish gigs loaded in to mm3.

Performance is 'ok', its not great tho. I find MM 2 currently more usable for now. I have not tried any scripts yet.

My main problem with mm3 is the debug dialog, if MM3 is 'busy' processing something (like expanding nodes on the 'locations' node is especially slow), then the debug dialog pops up and asks to send a log and/or terminate.

However in many cases if I uncheck to send log and and get rid of the dialog, then MM will eventually return.

Another thing is that add/rescan is much much slower than MM 2 with the same settings, which for reference are:

- infer OFF
- scan for art ON
- remove underscores ON
- analyze for dupes ON (yes unchecking this does speed up mm3, its still slow)
- ignore short songs OFF
- update track info ON
-- only for changed files ON
- remove unused artist/albums

------------------------------------

Is there a way to change the time out for the debug dialog to pop up?

[jiri]

Do you have any timings? Is it only when adding new tracks, or also when rescanning?

Jiri

[Teknojnky]

Both. It takes several (like 4+ over night hours) to rescan my entire library. The intial scans (it took several because there were corrupted embedded images that MM locked up on) took several nights worth.

Also, I have noticed that if there is a background process going, if you add new files to a monitored folder, that mm3 will also 'freeze' up for a few minutes (2-5 minutes it seems, I have not logged a specific time) but eventually it returns to normal operation.

Note also, I am currently NOT using auto-background analayze unknown volume. Until I confirm that MM writes the same replay tags as foobar/dbpoweramp r12, I will abstain from using it.

see post @ http://www.mediamonkey.com/forum/viewto ... 7307#77307 for details

[cmx]

I have about 96k files 775 GB....

P4 1.8 Ghz 1GB RAM

I quite often have CPU intensive software running with MM...so it needs to "play well with others"

Upon first starting in MM3 with volume leveling "on" it locked up computer...forget that....

I now have:
Infer -on
scan art -on
remove underscores -on
remove unused artists -on
remove unused albums -on

I think everything else is off....

A "primer" with all pertinent info concerning library "scanning" in one place would be appreciated by those of us with large libraries....the help files define things, but I don't feel like they "put it all together" as far as how MM "scanning/analyzing/synchronizing /etc..."functions. Any help along these lines in understanding this would be appreciated.

I have had to fully scan and rescan the full library several times for various reasons and testing...

I've noticed that scanning seems to drastically slow as time goes on almost as if there is a mem leak...for example I started a NEW scan from scratch and 8.5 hours later it had only done 65k...I then restarted MM as it had slowed to only several tracks per minute...and a new RESCAN finished the whole 96k in less than 3 hours....

Side-by-side the MM3 alpha 2 seems to operate loading tracks, searches, etc... as good or better than MM2 here...

*?*- Is getting all the tracks "read" pretty much just a function of my memory buss speed at this point?...even tho I have 1 GB RAM, I have very old 133 mem buss...

*?*One request concerning searches....there is a "lag", sometimes quite significant(up to 15-20 seconds), before tracks begin showing up and the "reading tracks" indicator is visible...it is intuitively uncomfortable...MM seems to be locked up or not responding...could there be a postive visual feedback that the search is indeed initiated and proceeding from the outset?

Without MM I couldn't practically utilize a library of this size...no other full featured player comes close.

The biggest new thing here is the M4A tag support...thank you... about 20% of my library is M4A and not having full access has been a problem. For me "tagging" fits with the "scanning" and large library issues.

Some big tagging issues concerning "tagging" here:

-all M4A tracks are labeled "1" here

-Compilations(whether MP3 or M4A that ARE correctly grouped by album/album artist in MM2, are NOT grouped in MM3, for example:

"100% Disco" (Please no comments on choice of example...LOL)seems to be tagged differently in MM3 than MM2. In MM2 it is correctly grouped by album and "Various" album artist....In MM3 "Various" as album artist does not seem to work...







Thanks for the great work.

[RandallSG]

I concur with previous comments, the "re-scan" is slow in MM3, particularly as compared to MM2. It took in excess of 14 hours to re-scan 24,000 tracks which were already in the MM3 library. The same re-scan of tracks in MM2 takes only a little more than 1 1/2 hours.

Not sure if it makes a difference, however, all of these tracks were M4A. These were the first tracks I loaded in MM3 Alpha 2 because I was excited to see if it would read the tag information.

I will provide more details on performance in a few days when the load is complete!

[Randall Grogan]

RovingCowboy, I should also note that MM is the only application running on the PC right now.

I will be curious to see if the application slows down as we approach the 100k mark.

[cmx]

I have increased my library to 120k files 940GB from 96k files 775 GB.

From my last post I changed the file structure somewhat when I added the new files...( Both MM2 and 3 also had version updates in the interim too...because the problem seems to exist the same on both I'm assuming first, that the problem has something to do with me and not a program change....)

PROBLEM: MM2 and MM3 now take up to 7 minutes to "Read All Tracks" previously it was around a minute...other searches and other incremental database queries seem to take proportionally longer too.

...now I essentially lock up th computer when MM2/3 is reading tracks....

WHAT happened?

I changed file structure a bit and moved the majority of files to single partition across several drives via a Lacie 1TB "drive", I also combined things under few parent directories...

....previously I had the files spread over several smaller partition over several drives....

IS there a "better" files structure that MM can handle more efficiently?

Experimenting with different library configurations is not easy because re-scanning can load the CPU for 10 -12 hours at a time. It is NOT a trivial matter rescanning 1TB of data.

COULD you please descibe exactly how to create completely seperate/discrete MM "libraries"....I would like to try to recreate a library with the files structure I had previously...it seemed to work very well...at least MUCH better than things are going now. With the current conditions here MM very difficult to use.

Any help in understanding how things work at this level would be appreciated.