big syncing, many crash, woe :(

Get answers about the current version of MediaMonkey 5

Moderator: Gurus

Peke
Posts: 17446
Joined: Tue Jun 10, 2003 7:21 pm
Location: Earth
Contact:

Re: big syncing, many crash, woe :(

Post by Peke »

Barry4679 wrote: Mon Oct 03, 2022 1:35 am
Peke wrote: Sun Oct 02, 2022 7:26 pm external 1TB SSD set as sync device totaling 982GB.
Interesting ... thanks.
was connection to your PC wired or via WiFi?
External HDD connected over USB 2.0 port AN dset as sync device in Devices and Services, no device involved. I only have <64GB SDcards and unless I use WiFi for subsequent syncs (do not sync more than 5-10Min) I use SDcard Reader to do initial one, no device connection.
Barry4679 wrote: Mon Oct 03, 2022 1:35 am power off the target device during the sync
FYI I tried Disconnecting HDD during Sync and afterwards I needed to Format it again due the partition corruption.
Barry4679 wrote: Mon Oct 03, 2022 1:35 am What is a deal breaker with MM5 sync is that it is not robust enough to handle a situation when something goes wrong during the sync.
For me it continue where it left of especially on WiFi.

Sync process is straight forward. Power savings when you want sycn available always needs to be disabled. Especially if it is a Desktop PC or Laptop constantly connected to power. Also Wired connection from PC to router is recommended.

in 99% of cases default Windows settings do not work. At least in my experience.
Best regards,
Peke
MediaMonkey Team lead QA/Tech Support guru
Admin of Free MediaMonkey addon Site HappyMonkeying
Image
Image
Image
How to attach PICTURE/SCREENSHOTS to forum posts
feline1
Posts: 119
Joined: Fri Dec 11, 2015 5:36 am

Re: big syncing, many crash, woe :(

Post by feline1 »

Peke wrote: Mon Oct 03, 2022 4:22 am Power savings when you want sycn available always needs to be disabled. Especially if it is a Desktop PC or Laptop constantly connected to power. Also Wired connection from PC to router is recommended.

in 99% of cases default Windows settings do not work. At least in my experience.
Yes, what I would find is that if there was any Windows Power Management sleep functions (or even Screen Savers!) it invariably makes a USB2.0 sync crash. I have disabled all that stuff on my desktop PC.

When I have time, I will try to install the debug version of the latest MMW5 version and send you some logs, thank you :)

Currently MMW5 seems to be re-syncing my entire library, copying every single file across again from scratch, even though I have set it to 'enable sync mask', so it should not be copying the files that are already present. It is about 15 hours into the process so I am not going to interrupt it yet ;)
feline1
Posts: 119
Joined: Fri Dec 11, 2015 5:36 am

Re: big syncing, many crash, woe :(

Post by feline1 »

Rob_S wrote: Sun Oct 02, 2022 10:35 pm
Not to say that it shouldn't but it tries to do so many things like analyze volume levels, and convert unsuitable formats, and etc depending on your settings, some times back to basics can get the job done, if that is the goal.

If you just want a complete file for file and folder for folder sync, fast and hassle free...
In my case, I am not asking MediaMonkey to do all that much during the sync -
but yes, there are some format conversions (e.g. things that are WAVs in my source database, and get written to my Walkman as FLAC), and there is maybe 5% of my database that does not get synced, because they are duplicate tracks (e.g. I might have ripped a CD single, and some of the tracks on it are identical to the parent album, so to save space on my Walkman, I tell MediaMonkey not to sync those ones)
Barry4679
Posts: 2398
Joined: Fri Sep 11, 2009 8:07 am
Location: Australia
Contact:

Re: big syncing, many crash, woe :(

Post by Barry4679 »

Peke wrote: Mon Oct 03, 2022 4:22 am Sync process is straight forward. Power savings when you want sycn available always needs to be disabled. Especially if it is a Desktop PC or Laptop constantly connected to power. Also Wired connection from PC to router is recommended.
My PC is wired to the router.
I mentioned Power Saving interruption just as an example.
In my case the issues came as a result of MM5 crashing while syncing.
I did not experience any file or directory corruption ... only a problem were MM5 didn't know what it had done, and seemed unable to re-initialise itself without the deletion of a lot of transferred files, and requiring a lot of unnecessary re-transfer.

I was testing sync to Dropbox ... so very lengthy sync times due to Internet constraints.
MM5 crashes often. Especially if you are using it do something out of your normal patterns of use.
I do some testing of MM5 ... or also try to replicate some issues that people report here. I experience crashes often.

Previously I had successfully used Google Music Manager to transcode and sync approx 50,000 tracks up to Google Play Music. This took around a month to complete. I stopped and started it periodically to keep within the limits of my Internet plan. The process completed without other intervention.

I lost this data when Google shut down Play Music.
I tested to see if I could achieve the same thing with MM5 syncing to to Dropbox location.
I queued up about just half of my Albums where their AlbumArtist started with A...

MM5 was never able to complete even this task without crashing.
I suppose that I triggered some or all of these crashes while testing or concurrently using MM5.

But the tool is useless if you can't touch MM while it is working on a task that is going to take days to complete.

IMO MM5 sync is just a toy if it cannot handle unexpected interruption.
Peke wrote: Mon Oct 03, 2022 4:22 am in 99% of cases default Windows settings do not work. At least in my experience.
That seems bogus info. I would think that I have default Windows settings.
I experience minimal issues. I move a lot of data around to my NAS, and across the Internet via FTP and backups to the web.
Want a dark skin for MM5? This is the one that works best for me .. elegant, compact & clear.
feline1
Posts: 119
Joined: Fri Dec 11, 2015 5:36 am

Re: big syncing, many crash, woe :(

Post by feline1 »

I am currently about 20 hours into re-syncing my library.

Thanks to Peke's troubleshooting, I seem to have entered a new zone of behaviour 8)

Just to recap:
1. I reformatted the SD card in my walkman, and started re-syncing my entire 700GB library from scratch

2. About 2 days into the process (after several crashes and having to use Windows ScanDisk to repair the SD Card and re-continue the sync), it got stuck again with MediaMonkey thinking that about 500GB of synced files were now 'Other' files... and so declaring that there was no longer room on the SD Card to complete the sync.

3. As per Peke's advice, I deleted persistant.* files from %APP_DATA%\MediaMonkey...

4. This did not immediately clear the 500GB of 'other' files. However MediaMonkey allowed me to re-start the sync. It re-started from the beginning of the 700GB, and (even though 'enforce sync mask on existing files' is checked), it seems to be just over-writing all the existing transferred files on the SD Card...

5... as it goes, the size of the files it thinks are 'Other' on the disk seems to be gradually shrinking. I am optimistic that in a few more days :lol: it will have successfully completed the sync. The question is, after that, will it now recognise the existing synced files, or is it going to want to restart from the beginning every time?!

(I will have to wait until it finishes before installed a debug build and capturing logs... )
Peke
Posts: 17446
Joined: Tue Jun 10, 2003 7:21 pm
Location: Earth
Contact:

Re: big syncing, many crash, woe :(

Post by Peke »

Hi,
Why don't you simply takeout SDCard and sync over SDCard Reader?

https://www.amazon.com/s?k=portable+sd+ ... lectronics
Best regards,
Peke
MediaMonkey Team lead QA/Tech Support guru
Admin of Free MediaMonkey addon Site HappyMonkeying
Image
Image
Image
How to attach PICTURE/SCREENSHOTS to forum posts
feline1
Posts: 119
Joined: Fri Dec 11, 2015 5:36 am

Re: big syncing, many crash, woe :(

Post by feline1 »

Cos I don't have an SD card reader and I don't want to have to dismantle my Walkman every time I wanna sync it. 1TB SD cards are expensive and probably don't want to mechanically re-inserted several times a week for 5 years :lol:
Peke
Posts: 17446
Joined: Tue Jun 10, 2003 7:21 pm
Location: Earth
Contact:

Re: big syncing, many crash, woe :(

Post by Peke »

Hi,
That is done only once, main problem is that device connection is often 3-5 times slower than SDcard readers.

What actual device you have You said Sony walkman?
Best regards,
Peke
MediaMonkey Team lead QA/Tech Support guru
Admin of Free MediaMonkey addon Site HappyMonkeying
Image
Image
Image
How to attach PICTURE/SCREENSHOTS to forum posts
feline1
Posts: 119
Joined: Fri Dec 11, 2015 5:36 am

Re: big syncing, many crash, woe :(

Post by feline1 »

I have a Sony NW-ZX100HN.

Not sure what you mean about only having to use then card reader once?
Rob_S
Posts: 868
Joined: Wed Mar 10, 2021 1:53 pm

Re: big syncing, many crash, woe :(

Post by Rob_S »

I think he means once you get it properly and completely loaded up, any future syncs would only involve transferring a few files, thus easily doable over USB.

This whole issue puzzles me, so my question(s) to DEVS here is, does MM not check to see (and index) what actual files are located on the device, as opposed to some index it might keep of what it thinks was once transferred?

If so, this should enable restart from any point in the sync process, as it simply checks what is there, and only transfers files of newer date or missing??

If otherwise, how would it know if any files have been accidentally deleted, and need to be replaced?

The issue of "other" files residing on the destination, but not identified as existing tracks is puzzling, can anyone explain why that might occur?? Should these not first be indexed to see if they are valid tracks??

So, can somebody explain the basic logic that is used in the sync process, it might help people figure out what to do better.

This also relates to my suggestion of manually transferring the bulk of the files, and then use sync for topups.
Using 5.1 LATEST alpha or beta build on Windows 10, HP laptop, managing 13k tracks
feline1
Posts: 119
Joined: Fri Dec 11, 2015 5:36 am

Re: big syncing, many crash, woe :(

Post by feline1 »

Rob_S wrote: Mon Oct 03, 2022 4:18 pm I think he means once you get it properly and completely loaded up, any future syncs would only involve transferring a few files, thus easily doable over USB.
Ah OK - but unfortunately, this padt couple of years I have been finding that my library only last on my Walkman for 2 or 3 months before MediaMonkey gets confused and starts insisting that the 700GB of existing synced files are now 'other', and so we're back to square one!
Peke
Posts: 17446
Joined: Tue Jun 10, 2003 7:21 pm
Location: Earth
Contact:

Re: big syncing, many crash, woe :(

Post by Peke »

Hi,
Main problem is that Sony NW-ZX100HN is not an Android device, it is bit older device I see Last firmware is from 2015, Which most likely means it at max utilize full USB 2.0 bandwidth (480Mbit/60MBytes) which is in real life maxed at 20MBytes/s often 3-5MB/s which means at max speed means 3-4s per track.
On that you add Sony file handling which means more like 4-5s per track. If we calculate that the max transfer rate for 1000 tracks is 5000s eg. almost 1.5h :( that means 10k of tracks take 15h and in 99% of cases the larger number of tracks are on device then device is slower to handle transfer and update library, so time taken to device announce "send me next file" is longer and longer.

On other side using USB 3.0 Card reader should do the same job 10x faster and there will be no slowdowns due the fact that there will be no Device library update,...
Best regards,
Peke
MediaMonkey Team lead QA/Tech Support guru
Admin of Free MediaMonkey addon Site HappyMonkeying
Image
Image
Image
How to attach PICTURE/SCREENSHOTS to forum posts
feline1
Posts: 119
Joined: Fri Dec 11, 2015 5:36 am

Re: big syncing, many crash, woe :(

Post by feline1 »

Well this is true, but I actually bought the device in 2015 (as a replacement for a stolen iPod!) And that was when I first bought a gold licence for MediaMonkey (as a replacement for iTunes)

So I have been using this Walkman happily for over 7 years now, it still works fine... it just seems to be MMW which is increasingly unstable :(
I would have no problem waiting a few days to do the initial sync of the library - it is just the fact that MM keeps corrupting it every few months is the problem
Barry4679
Posts: 2398
Joined: Fri Sep 11, 2009 8:07 am
Location: Australia
Contact:

Re: big syncing, many crash, woe :(

Post by Barry4679 »

Rob_S wrote: Mon Oct 03, 2022 4:18 pm This whole issue puzzles me, so my question(s) to DEVS here is, does MM not check to see (and index) what actual files are located on the device, as opposed to some index it might keep of what it thinks was once transferred?

If so, this should enable restart from any point in the sync process, as it simply checks what is there, and only transfers files of newer date or missing??

If otherwise, how would it know if any files have been accidentally deleted, and need to be replaced?

The issue of "other" files residing on the destination, but not identified as existing tracks is puzzling, can anyone explain why that might occur?? Should these not first be indexed to see if they are valid tracks??
I am not a DEV, but I can guess at what is happening.

I suspect that what happens is that MM deletes what is old at the target, and then it transfers what is new, and then it builds a a db describing the known files that it is responsible for at the target. This gets stored at the target location.

When the sync, or MM, or your PC or the transport medium crashes, this last step doesn't happen, so the picture is out of date. There are probably snapshot updates during a big sync batch, but this still not enough for large sync tasks .. as reported here, and in a couple of earlier similar incidents .. mine included.

Without more defensive planning this is not so easy for MM to recover from. ie:
  • who put those files at the target location? Is it ok for MM to assume management of them? ... maybe they are a recording of some happy family event ... MM should not touch those
  • where are all the files that MM is managing? ... maybe you changed the sync-to location at some time, so there could potentially be files at multiple locations
  • what are those files at the target location? ... where do they fit into the MM db ... maybe you retagged those tracks at home after transferring them, ... maybe you did a db rebuild, which would have change all the album ids
  • how did you select those files to be transferred? .. maybe you selected them from your PartyMusic collection, using its Album index ... or maybe you selected them from the NewMusic collection, using the Genre index .. so how|where does MM
    know which checkbox they relate to in the "sync list" (aka "Library Content") panel in the Devices&Services node?


It really needs the up-to-date db at the target location, but a completely up-to-date version of that database does not exist

More defensive planning could keep an up-to-date copy of your main PC, and then your main PC could recognise that the target view is stale, and it could complete the missing step from the prior Sync, which failed.

Also I think that there is something that is fundamentally not up to the task in the MM sync mechanism.

There is a single sync control by which you:
  1. select files to be sync'd to the target ... nice to use: lots of choices: collections and each of their sub-nodes .. really great to use
  2. [and also use when housekeeping what you have sync'd .. ie. what you delete .. this is horrible to use, because you have to remember how|where you initially selected it?/list]

    More info here, but it is a long (boring?) read

    It is like "life" .. the early version was just a sac, with just the one opening.
    Luckily we evolved to have both en entrance, and exit, to our digestive system. :D
    I think that MM5 would benefit by evolving.

    If they were to provide both:
    1. a sync selection control ... which could also optionally be used for housekeeping
      • and also a sync housekeeping control ... that is optional, but is only used for deletions ... it is a much flatter structure, that doesn't have all multitude of Collections and nodes
    That would resolve issues raised in the above Mantis report ... and I think that what they had to do in the MM db to support this would assist a lot with the issues postulated above.
Want a dark skin for MM5? This is the one that works best for me .. elegant, compact & clear.
Peke
Posts: 17446
Joined: Tue Jun 10, 2003 7:21 pm
Location: Earth
Contact:

Re: big syncing, many crash, woe :(

Post by Peke »

feline1 wrote: Mon Oct 03, 2022 6:13 pm I would have no problem waiting a few days to do the initial sync of the library - it is just the fact that MM keeps corrupting it every few months is the problem
I have never seen that for past 5 years and MMA <-> MMW/MM5 Sync at least 3-5 times a day eg. each time I park the car newr my home WiFi and it sync to 64GB SDCard around 5k of tracks and 40+ Playlists. My SDcard is working normally no corruption. I am suspecting something in relation to Sony handling such large SDcard, as even it is 2022 many devices struggle with such SDcards and large number of files.

NOTE that in 2015 1TB SDcards did not exist and controller chip technology changed a lot meantime. Have seen devices that can't even access/format such cards. Like my dashcam for example that struggle with 256/512 SDCards and I need to format it every few months to make them work again.
Best regards,
Peke
MediaMonkey Team lead QA/Tech Support guru
Admin of Free MediaMonkey addon Site HappyMonkeying
Image
Image
Image
How to attach PICTURE/SCREENSHOTS to forum posts
Post Reply