Smart Playlists - duplicate track removal = smart pruning

To discuss development of addons / skins / customization of MediaMonkey.

Moderators: jiri, drakinite, Addon Administrators

telecore
Posts: 57
Joined: Thu Jan 28, 2021 10:20 pm

Smart Playlists - duplicate track removal = smart pruning

Post by telecore »

In MM4, I had written a useful script that removes tracks with duplicate artists and titles from playlists. It worked by searching for matching artist and titles within the same playlist (with a significant amount of string processing for cleanup). The output was a manual playlist copy of the original (original=smart or manual) with the same name except for an asterisk at the end. For me, this was extremely useful for removing duplicate tracks from playlists that may be present on different albums - without having to delete one of the copies of the track, therefore, keeping all albums intact. The "pruned" playlist is what ends up being used for playback and synced to devices.

I would really like to duplicate this in MM5, however, it would have to be re-written from scratch as scripting has been totally changed. I have a programming background, but I really had to spend a lot of time writing the original MM4 script by "hacking" examples from others and I really don't have the time to re-do this (I tried a few months ago).

I think this could be a useful built-in feature of MM5, to add a "smart pruning" capability for playlists - the existing "remove duplicates", for manual playlists only doesn't really help - is this something that the developers would consider adding as a feature?
Ludek
Posts: 4945
Joined: Fri Mar 09, 2007 9:00 am

Re: Smart Playlists - duplicate track removal = smart pruning

Post by Ludek »

Hi, adding script/addon like this should be quite easy.

In actions.js there is already an action for removing playlist duplicates by track.id, i.e. this code:

Code: Select all

 playlistRemoveDuplicates: {
            title: function () {
                return _('Remove duplicates');
            },
            icon: 'remove',
            visible: function () {
                if (!window.uitools.getCanEdit())
                    return false;
                else {
                    var pl = resolveToValue(this.boundObject);
                    return (pl.parent != undefined && !pl.isAutoPlaylist); // to exclude root playlists node and auto-playlists
                }
            },
            execute: function () {
                var pl = resolveToValue(this.boundObject);
                var list = pl.getTracklist();
                list.whenLoaded().then(() => {
                    list.modifyAsync(() => {
                        var hasher = {};
                        listForEach(list, (track, idx) => {
                            if (hasher[track.id])
                                list.setSelected(idx, true); // duplicate
                            hasher[track.id] = true;
                        });
                        pl.removeSelectedTracksAsync(list);
                    });
                });
            }
        },
All that needs to be changed are the lines around hasher[track.id] like this:

Code: Select all

if (!hasher[track.title])
     newList.add(track); // not a duplicate
hasher[track.title] = true;
And create new playlist like this and put the new list there:

Code: Select all

var newplaylist = app.playlists.root.newPlaylist();
newplaylist.name = pl.name + ' (filtered)';
newplaylist.commitAsync().then(function () {
                    newplaylist.addTracksAsync(newList);
});

So in your addon create actions_add.js and put following there:

Code: Select all

actions.playlistRemoveDuplicatesByTitle = {
    title: function () {
        return _('Remove duplicates by title');
    },
    hotkeyAble: true,
    icon: 'remove',
    visible: function () {
        if (!window.uitools.getCanEdit())
            return false;
        else {
            var pl = resolveToValue(this.boundObject);
            return (pl.parent != undefined); // to exclude root playlists node
        }
    },
    execute: function () {
        var pl = resolveToValue(this.boundObject);
        var list = pl.getTracklist();
        var newList = app.utils.createTracklist();
        list.whenLoaded().then(() => {
                var hasher = {};
                listForEach(list, (track, idx) => {
                    if (!hasher[track.title])
                        newList.add(track); // not a duplicate
                    hasher[track.title] = true;
                });
                var newplaylist = app.playlists.root.newPlaylist();
                newplaylist.name = pl.name + ' (filtered)';
                newplaylist.commitAsync().then(function () {
                    newplaylist.addTracksAsync(newList);
                });
        });
    }
};

And then put to to context menu of playlist by creating viewHandlers_add.js and puttin this code:

Code: Select all

nodeHandlers.playlist.menuAddons = nodeHandlers.playlist.menuAddons || [];
nodeHandlers.playlist.menuAddons.push(function (node) {
    if (node && node.dataSource) {
        return [{
            action: bindAction(window.actions.playlistRemoveDuplicatesByTitle, () => {
                return node.dataSource;
            }),
            order: 40,
            grouporder: 10,
        }];
    };
    return [];
});
Last edited by Ludek on Fri Sep 17, 2021 8:55 am, edited 3 times in total.
Ludek
Posts: 4945
Joined: Fri Mar 09, 2007 9:00 am

Re: Smart Playlists - duplicate track removal = smart pruning

Post by Ludek »

The working script is here: https://www.dropbox.com/s/td1t3albp2jgr ... .mmip?dl=0
feel free to adjust/tune the code ;-)
telecore
Posts: 57
Joined: Thu Jan 28, 2021 10:20 pm

Re: Smart Playlists - duplicate track removal = smart pruning

Post by telecore »

Thanks! - I just saw this and downloaded

Update: Got it working well - although it is a different approach than my previous script (this new one uses hashing), it is working about 99% the same as my MM4 script for removing duplicate artists/titles - Thanks again!
telecore
Posts: 57
Joined: Thu Jan 28, 2021 10:20 pm

Re: Smart Playlists - duplicate track removal = smart pruning

Post by telecore »

I have been trying to enhance this script and ran into a problem - I want to create a copy of a smart playlist and have the script wait until that copy is ready before having the program proceed - I am trying to use playlist.createCopyAsync(); - I have a background in real-time C programming for embedded systems and am trying to catch up on javascript, promises, etc.

Given a playlist, how do I create the copy and have the script for it to be ready?

This doesn't seem to work (pl= the playlist to process and it is already available at this point)

var pl_sort = pl.createCopyAsync().then(() => {
// actions here
});

(Then I want to get the tracks, re-sort and use this new playlist as the one to run my previous script on so that it selects duplicates with the first track on the list being preferred)

Thanks in advance for any replies!
drakinite
Posts: 965
Joined: Tue May 12, 2020 10:06 am
Contact:

Re: Smart Playlists - duplicate track removal = smart pruning

Post by drakinite »

Promises and async/await are a bit counter-intuitive until you get used to them. The variable type that's returned by pl.createCopyAsync() is actually a Promise, and the way to get the actual playlist copy is either via "await" or by taking a parameter from the ".then()" callback:

You don't have to do it in multiple lines, but hopefully this helps clear the confusion as to which variables are what:

Code: Select all

pl2promise = pl.createCopyAsync();
pl2promise.then((pl2) => {
    // do stuff with pl2
    console.log(pl2.title);
    console.log(pl2.asJSON);
})
here's an async/await way of doing the same thing:

Code: Select all

async function domystuff() {

    var pl2 = await pl.createCopyAsync();
    console.log(pl2.asJSON);
}

domystuff();
https://developer.mozilla.org/en-US/doc ... c_function
Image
Student electrical-computer engineer, web programmer, part-time MediaMonkey developer, full-time MediaMonkey enthusiast
I uploaded many addons to MM's addon page, but not all of those were created by me. "By drakinite, Submitted by drakinite" means I made it on my own time. "By Ventis Media, Inc., Submitted by drakinite" means it may have been made by me or another MediaMonkey developer, so instead of crediting/thanking me, please thank the team. You can still ask me for support on any of our addons.
telecore
Posts: 57
Joined: Thu Jan 28, 2021 10:20 pm

Re: Smart Playlists - duplicate track removal = smart pruning

Post by telecore »

Thanks - that worked - would you mind helping with the code to re-sort the playlist below?
...
// create copy of the playlist
var pl2promise = pl.createCopyAsync();
//---------------------------------------------------------------------------------
pl2promise.then((pl_sort) => {
pl_sort.name = pl.name + ' (sorted)';
pl_sort.parent = pl;

var tracks_sort = pl_sort.getTracklist();
tracks_sort.whenLoaded().then(() => {
tracks_sort.beginUpdate();
tracks_sort.setSortRule("Rating Z..A; Bitrate Z..A; Length Z..A;");
tracks_sort.endUpdate();

pl_sort.beginUpdate();
pl_sort.reorderAsync(tracks_sort);
pl_sort.endUpdate();

pl_sort.commitAsync();

The playlist is created but does not get sorted - still having difficulties with async programming in js - sometimes bad code will lock up MM5 -
drakinite
Posts: 965
Joined: Tue May 12, 2020 10:06 am
Contact:

Re: Smart Playlists - duplicate track removal = smart pruning

Post by drakinite »

I did some digging and your question has led me to learn a lot about sorting and auto-playlists that I did not before. Several notes:

1. Documentation on tracklist sorting is terribly lacking. I've updated the documentation, but it'll be a bit before it's live on the site. Here's a copy of setAutoSortAsync and setSortRule that I've written:

setAutoSortAsync:
ASYNCHRONOUSLY sorts the list with the given sorting rule and updates the auto-sort rule.
To specify sort direction, append ' ASC' or ' DESC' to the tag name and separate them by semicolons.
Fields are NOT case sensitive (e.g. 'artist', 'Artist', and 'ArTiSt') are all valid, but sort direction IS case sensitive ('ASC' and 'DESC' are valid; 'asc' and 'desc' are not)

Valid examples:
list.setAutoSortAsync('title');
list.setAutoSortAsync('artist; title');
list.setAutoSortAsync('rating DESC; title ASC');
list.setAutoSortAsync('Rating DESC; TITLE;');


Invalid examples:
list.setAutoSortAsync('tagThatDoesNotExist');
list.setAutoSortAsync('rating desc; title asc');


setSortRule:
SYNCHRONOUSLY sorts the list with the given sorting rule and disables auto-sort.
To specify sort direction, append ' ASC' or ' DESC' to the tag name and separate them by semicolons.
Fields are NOT case sensitive (e.g. 'artist', 'Artist', and 'ArTiSt') are all valid, but sort direction IS case sensitive ('ASC' and 'DESC' are valid; 'asc' and 'desc' are not)

Valid examples:
list.setSortRule('title');
list.setSortRule('artist; title');
list.setSortRule('rating DESC; title ASC');
list.setSortRule('Rating DESC; TITLE;');


Invalid examples:
list.setSortRule('tagThatDoesNotExist');
list.setSortRule('rating desc; title asc');


2. You don't need to do beginUpdate() and endUpdate() in this case. It is useful when we have UI controls who have listened to certain events from their dataSources. BUT since this is a brand-new clone, and you have not listened to events (e.g. app.listen() or localListen()), it'll have no impact (description of beginUpdate: Lock object to update state. Events are not called when in update state.)

3. You'll actually need different code for auto-playlists and regular playlists. You were using reorderAsync() correctly for a regular playlist, but auto-playlists use a more complex sorting method and reorderAsync() does not work for them. (I've also updated the docs to note this)
Auto Playlists use a QueryData object to handle their sorting. You can check searchEditor.js and playlistHeader.js for how these QueryData objects are created and updated. Also, the property isAutoPlaylist can be read to check if a given playlist is an auto-playlist AND updated in case you want to switch its type.

4. It's better to use tracks_sort.setAutoSortAsync() to avoid blocking the main UI thread.

5. Because of all the async functions required, it'll be much easier to use an async function and use await each in order. (Otherwise, you'd be doing a lot of .then()s and lots of annoying indentation, a.k.a. "callback hell")

All together:

Code: Select all

pl.createCopyAsync().then(async (pl_sort) => {
    
    pl_sort.name = pl.name + ' (sorted)';
    pl_sort.parent = pl;
    
    // Do QueryData shenanigans if it's an auto-playlist
    if (pl_sort.isAutoPlaylist) {
        let queryData = await app.db.getQueryData({ category: 'empty' }); // Create a new QueryData object
        queryData.loadFromString(pl_sort.queryData); // Transfer all its properties from the playlist (essentially creating a copy)
        queryData.setSortOrders([ // Set its sort order: this time it's an array of objects insteada of a string
            {
                name: 'Rating',
                ascending: false,
            },
            {
                name: 'Bitrate',
                ascending: false,
            },
            {
                name: 'Length',
                ascending: false,
            },
        ]);
        pl_sort.queryData = qd.saveToString(); // Update the playlist's queryData 
        // (Note: TPlaylist.queryData isn't actually stored as a string. Internally, it uses saveToString() as a getter and loadFromString() as a setter.)
        pl_sort.commitAsync(); // Commit the playlist
        pl_sort.notifyChanged('tracklist'); // to live update tracks -- is listened e.g. by viewHandlers.tracklistBase.onShow
    }
    // otherwise, use the tracklist sort method
    else {
        let tracks_sort = pl_sort.getTracklist();
        await tracks_sort.whenLoaded();
        await tracks_sort.setAutoSortAsync("Rating DESC; Bitrate DESC; Length DESC;");
        await pl_sort.reorderAsync(tracks_sort);
        pl_sort.commitAsync();
    }
});
telecore wrote: Mon Sep 19, 2022 1:33 pm still having difficulties with async programming in js - sometimes bad code will lock up MM5 -
Last note is unrelated to playlists and tracklists: MM5 will often lock up and crash if you do something wrong with a native Delphi object or method (All of the objects we've been messing with here are native MM objects). When in doubt, check the documentation, and if the documentation isn't clear, don't hesitate to ask for us to clarify (and update the documentation to be more clear).
Image
Student electrical-computer engineer, web programmer, part-time MediaMonkey developer, full-time MediaMonkey enthusiast
I uploaded many addons to MM's addon page, but not all of those were created by me. "By drakinite, Submitted by drakinite" means I made it on my own time. "By Ventis Media, Inc., Submitted by drakinite" means it may have been made by me or another MediaMonkey developer, so instead of crediting/thanking me, please thank the team. You can still ask me for support on any of our addons.
telecore
Posts: 57
Joined: Thu Jan 28, 2021 10:20 pm

Re: Smart Playlists - duplicate track removal = smart pruning

Post by telecore »

Thank you for the in-depth information - after some major re-writing, the script is now working better than its MM4 counterpart
drakinite
Posts: 965
Joined: Tue May 12, 2020 10:06 am
Contact:

Re: Smart Playlists - duplicate track removal = smart pruning

Post by drakinite »

Excellent to hear! :grin:
Image
Student electrical-computer engineer, web programmer, part-time MediaMonkey developer, full-time MediaMonkey enthusiast
I uploaded many addons to MM's addon page, but not all of those were created by me. "By drakinite, Submitted by drakinite" means I made it on my own time. "By Ventis Media, Inc., Submitted by drakinite" means it may have been made by me or another MediaMonkey developer, so instead of crediting/thanking me, please thank the team. You can still ask me for support on any of our addons.
Peke
Posts: 17446
Joined: Tue Jun 10, 2003 7:21 pm
Location: Earth
Contact:

Re: Smart Playlists - duplicate track removal = smart pruning

Post by Peke »

telecore wrote: Wed Sep 21, 2022 2:00 pm Thank you for the in-depth information - after some major re-writing, the script is now working better than its MM4 counterpart
Wasn't that purpose of MM5 ;) as it means that years of developing was worth.
Best regards,
Peke
MediaMonkey Team lead QA/Tech Support guru
Admin of Free MediaMonkey addon Site HappyMonkeying
Image
Image
Image
How to attach PICTURE/SCREENSHOTS to forum posts
OliverBradley
Posts: 2
Joined: Wed Dec 27, 2023 11:33 pm

Re: Smart Playlists - duplicate track removal = smart pruning

Post by OliverBradley »

Is this script available for download? Thanks!
Post Reply