Proposal for Auto Download Redesign

[peakvalleytech]

I believe the primary use of global episode cache is used to prevent an app from downloading an excessive number of new episodes. The problem is that when limit is reached the app will start deleting episodes globally. And since auto download fetches the newest episodes globally, feeds will compete for each other over downloads (A feed with many new episodes will beat a feed with only one) and could result in download starvation of other feeds.

Many have complained about this and many have proposed suggestions to fix it. Many have suggested a feed specific auto download settings. After some thought and reading through the issues, I have reached the same conclustion. Here, I will explain why I support this and my specific proposal.

A global setting is not entirely wrong and many apps have it, like ourselves. But it only works when it is assumed that no download limit exists. In this case, there is no starvation. However, assuming there is a set limit, things get a little complicated. It seems that AP originally didn’t have a episode limit and was added later. Overtime other features were added without a proper redisgn leading to our current need for a complete rework.

Our current solution to this problem seems to be auto deleting episodes to help make room. But this only minimizes the problem and the problems mentioned above could occur in certain cases. Even more important, the implementation was ill designed, since we already have a type of auto deletion elsewhere in the app, leading to confusion and user complaints (#1486, #3063). After many more complaints, tt has been decided to rework the entire feature. @Bytehamster has requested voluteers and this is my reply.

To come up with a solution, I considered the filed issues under the mlestone, ‘Rework auto download’
I believe the most useful suggestion is ‘the keep last N episodes suggestions’ and used it as the basis for my solution.
My solution is essentially to implement the above suggestion by removing the global episode download limit and replacing it with a feed specific setting ‘keep last N episodes’ (#2077)

With this feed specific setting to limit the number of episodes to keep, a global setting to limit episodes serves no purpose. Besides it will help in reducing confustion, and eliminates many issues automatically (#5236, #4795)

Below, I’ve added my own specific implemntation ideas and I’m open to other suggestions.

This setting will not be a a hard limit like the global setting, but something a soft limit. Soft meanig that the will not eforce that epsidoes passed N will not be allowed. The setting only makes sure the last N episodes are downloaded. No assumtions will be made about episodes past N. This will be entirely managed by a seperate episode auto delete feature. And will work nicely to solve #1856.

In the current design, an episode clean up algorithmm is used by auto download service to make room for new episodes. With the new design, this will be decaupled, as suggested by @Bytehamser, from auto download. This means auto deleting will now only be found and set under one location (maybe storage settings). And code will be cleaner and better able to handle new features…

I came to this approach for a ‘soft limit’ after realizing that coupling auto download and auto delete is indeed a very bad idea. It makes implementing keep N episodes very complicated and is not necssary if we use this ‘soft limit’

The auto delete setting will allow the user to select the cretieria for deleting episodes. It will implment many of the suggestions regarding finer control of auto delete (#1275, #746). With feed specific episode limit, auto delete can be simplified to only deleting deleting episodes outside of the N epsidoes definied by feed settings. There is no longer a need to clean up episodes to help auto download work better and no starvation of feeds.

The only immediate issues I can see from this design is the following:
a. Not having easy way to determine the number of download limit for all podcasts.
b. Having to micro manage auto downloads

Solutions to
a. Improve stats, such as storage stats which will display the number of episodes on device and storage in GB
b. Have a defaultl N setting and a retain the gobal setting to turn on auto download for all feeds

I hope I have considered all the possible issues and comments in the milestone. Whether this solution is used, I hope this can get the ball rolling on this milestone.

I apologize for the length. I wanted to be as clear as possible.

[peakvalleytech]

Hello. Any updates on this yet? I know this is a milestone and I would really like to help to get it done.

[RTVance]

I would like to have the option to have the podcasts downloaded to the /Downloads/… or /Podcasts I dislike having to find the folder I am using to save files in.
Since self-cleaning never seems to work just right in any of the apps I’ve used over the past years I have had to get ‘clean’ gigs of files.

As for getting started
It seems to me that you start working on it submit upstream, in chunks, and if it isn’t accepted you fork it. . . The Opensource Model.

[keunes]

Thanks for initiating the discussion on this @peakvalleytech. It’s quite a text, so I’ll have a look at it when I have a bit more time (hopefully soon)

[ByteHamster]

Sorry, it took a while because the text is really long and because the redesign is coupled with quite a few difficult decisions.

@peakvalleytech

Overtime other features were added without a proper redisgn leading to our current need for a complete rework.

Yup, that’s the main problem with auto download

@peakvalleytech

replacing it with a feed specific setting ‘keep last N episodes’

I think the reworked setting should be in some way backwards compatible. Otherwise, we risk a bunch of complaints by users. So there should be a way to set it up that is “close enough” to the current behavior (whatever that means).

@peakvalleytech

Soft meanig that the will not eforce that epsidoes passed N will not be allowed. The setting only makes sure the last N episodes are downloaded. No assumtions will be made about episodes past N. This will be entirely managed by a seperate episode auto delete feature.

Sounds good to me but I think there still needs to be some kind of global setting. We also have a request somewhere to limit the total auto-downloaded episodes by storage space, which doesn’t really make sense on a per-subscription basis.

@peakvalleytech

This means auto deleting will now only be found and set under one location (maybe storage settings).

I like that it is decoupled. Still, I wonder how it would work in the UI. Would there be two independent per-subscription settings to set the numbers for auto download and auto delete? I don’t really see a use case for two different numbers there. Then we would need to come up with a good name, though.

@peakvalleytech

Have a defaultl N setting and a retain the gobal setting to turn on auto download for all feeds

If you are subscribed to, say, 10 feeds but 5 of them are “dead” (no longer produce new episodes), setting the default N could be confusing. I like the idea though. Not sure how to tweak it for clarity.

@peakvalleytech

It will implment many of the suggestions regarding finer control of auto delete

When auto delete is set to something like “episodes older than x days, independently of listened or not”, this would overwrite auto-download, right? So even if it is in the top N episodes, it doesn’t get downloaded. When now increasing the N, we need to make sure that we don’t download all N episodes and immediately delete them afterwards (that’s the advantage of having the features coupled: they work together. Still, I think it is better to decouple)

[peakvalleytech]

That’s not the only text that is long. I will reply in detail soon.

[peakvalleytech]

@ByteHamster

I think the reworked setting should be in some way backwards compatible. Otherwise, we risk a bunch of complaints by users. So there should be a way to set it up that is “close enough” to the current behavior (whatever that means).

That should be kept in mind. Thanks for bringing this up. However, I don’t know if it is possible to be backwards compatible. The whole point the this rework is to overhaul the entire feature, right? And we will be adding features, not replacing them. Currently, there is nothing close to a feed specific ‘keep last N’ setting.

[peakvalleytech]

@ByteHamster

Sounds good to me but I think there still needs to be some kind of global setting. We also have a request somewhere to limit the total auto-downloaded episodes by storage space, which doesn’t really make sense on a per-subscription basis.

Yes, this is basically the default setting. I don’t think we should have a limit for storage. It does’t really add much benefit. We will already have features to put constraints on downloads (keep N episodes and auto delete). We don’t need another one that will only add more development effort and complexity. Part of this new design was to circumvent some of the mentioned issues entirely.

[peakvalleytech]

@ByteHamster

I like that it is decoupled. Still, I wonder how it would work in the UI. Would there be two independent per-subscription settings to set the numbers for auto download and auto delete? I don’t really see a use case for two different numbers there. Then we would need to come up with a good name, though.

No, auto download will not have a setting for auto delete as it won’t handle auto delete. Auto delete will be one setting located in TBD.

On the surface (UI), it the two will be serperate. But underneath auto download may call auto delete to delete clean up episodes prior to auto download, like it currently does. However, it won’t be to ‘make room for autodownload.’

Auto download will only find the latest N episodes and download those. And N won’t necdessairly be the newest episodes. It will depend on user settings which users have requested such as chaning the order, including older episodes …etc.) Those N episodes will not be inluded in auto delete. Currently, we limit the downlaods using a gobal episode cache. This new design will use auto delete to manage downloads. Thus theyre is no need for an episode cache. Using auto delete, users can have better control than a episode download limit. For example they can set it to never delete episodes, which is same as unlimited cache. Or they can set it to only delete played episodes same as having a cache limit.

[peakvalleytech]

@ByteHamster

If you are subscribed to, say, 10 feeds but 5 of them are “dead” (no longer produce new episodes), setting the default N could be confusing. I like the idea though. Not sure how to tweak it for clarity.

I’m not sure what you mean. Could you clarify?

[peakvalleytech]

@ByteHamster

When auto delete is set to something like “episodes older than x days, independently of listened or not”, this would overwrite auto-download, right? So even if it is in the top N episodes, it doesn’t get downloaded. When now increasing the N, we need to make sure that we don’t download all N episodes and immediately delete them afterwards (that’s the advantage of having the features coupled: they work together. Still, I think it is better to decouple)

Sorry, I wasn’t clear enough. The N episodes won’t be included in auto delete. Auto delete will only delete episodes not in N. Also, we should improve the download screen.

[RTVance]

Continuing the discussion from Proposal for Auto Download Redesign:

@ByteHamster

@peakvalleytech

Have a defaultl N setting and a retain the gobal setting to turn on auto download for all feeds

If you are subscribed to, say, 10 feeds but 5 of them are “dead” (no longer produce new episodes), setting the default N could be confusing. I like the idea though. Not sure how to tweak it for clarity.

In regards to this, it would be a good idea to look at how other apps do this. Podcast Addict and Doggcatcher both have a ‘keep at most’ and ‘delete marked as listened to’ podcasts. They both use different words for the same settings.
Being that there isn’t a dialog in Podcast Addict explaining how it works it must make sense to his users… because he has the customer service skills of a walrus, leave him alone and you are fine, cause him to have to read his email at all beware the tusks.
I am personally responsible for several dialogs explaining what is ‘supposed’ to happen… but it works that way only in programmer logic, not in English language logic. It was easier to write a paragraph to explain, than change 2 words for clarity. (This kind of behavior is why I finally stopped using the app.)

@ByteHamster

@peakvalleytech

replacing it with a feed specific setting ‘keep last N episodes’

I think the reworked setting should be in some way backwards compatible. Otherwise, we risk a bunch of complaints by users. So there should be a way to set it up that is “close enough” to the current behavior (whatever that means).

To be honest, the way you have it set up now hasn’t made any sense to me; in all the apps I tested basically none of them had it set as a global files limit. Almost all well rated apps had an option to limit the total space dedicated to downloading podcasts. (I tested 10 or so apps in the least several months trying to find an app to replace Podcast Addict.)

@ByteHamster

@peakvalleytech

Soft meanig that the will not eforce that epsidoes passed N will not be allowed. The setting only makes sure the last N episodes are downloaded. No assumtions will be made about episodes past N. This will be entirely managed by a seperate episode auto delete feature.

Sounds good to me but I think there still needs to be some kind of global setting. We also have a request somewhere to limit the total auto-downloaded episodes by storage space, which doesn’t really make sense on a per-subscription basis.

How Doggcatcher and PA have done to address this issue is to use a priority-based system. The higher the Priority the more likely it is to download.

Presently you have the ‘Set Subscription Order’ option. So you are part way to implementing this.

Additionally though just because it has a higher priority should not mean you only download based on that. PA had the option to go by Priority, then Date or Duration or…

–
As a separate comment… please work on this. I am on the 4th time trying to get the downloads that should have been downloaded to download. Because I was having wifi problems I had 200+ files that all had less than 10mb. I had to go in and delete all the files in order to not queue 200 files by hand… needless to say, that would have been my final time using the app.
This is also why I want to option to have the podcasts in the Podcasts or Download folder.

[peakvalleytech]

@RTVance

In regards to this, it would be a good idea to look at how other apps do this. Podcast Addict and Doggcatcher both have a ‘keep at most’ and ‘delete marked as listened to’ podcasts. They both use different words for the same settings.

Yeah, sorry to hear about your experience with PA. I guess its UI is the only thing that is bad.

I’ve had actually look at the auto download features of other podcasts. PA is the only one that comes close to what we are looking at doing.

It allows users to set a limit (keep at most) for downloads. After looking at it closer, it seems to be a less flexible version the design of this proposal.

The key differentiator is that our design will be a lot less coupled. Our N setting will not be a limit but rather a predefined number of downloads. So instead of ‘keep at most’ it is more like ‘keep latest’, where latest is not necessarily the latest episodes, but latest as defined by user settings, which have been discussed previously (Download older episodes, change download order, etc).

It will act separately from auto delete. Auto delete will only apply to downloads not included in the N latest for a podcast. In this way, user will have more control over downloads. They can either set it to never delete episodes or delete depending on user settings( After played, keep favorite, keep not played, keep in queue, etc)

[peakvalleytech]

@RTVance

To be honest, the way you have it set up now hasn’t made any sense to me; in all the apps I tested basically none of them had it set as a global files limit. Almost all well rated apps had an option to limit the total space dedicated to downloading podcasts. (I tested 10 or so apps in the least several months trying to find an app to replace Podcast Addict.)

I agree, looking over the code, in regards to auto download, it is a complete mess. This will take a lot of work. Yeah, a global setting is the primary source of most of the current issues.

[peakvalleytech]

@RTVance

How Doggcatcher and PA have done to address this issue is to use a priority-based system. The higher the Priority the more likely it is to download.

Presently you have the ‘Set Subscription Order’ option. So you are part way to implementing this.

Additionally though just because it has a higher priority should not mean you only download based on that. PA had the option to go by Priority, then Date or Duration or…

I’m not sure what issue you are referring to. I assume you are talking about the current old design.

Yeah, a priority system has been brought up, but it only mitigates the issue. Eventually the same probblems would occur even with priority given a increase in podcasts. That is what a podcasts specific auto download would solve.

On the topic of priority-based. They’re is still some benifit as has been discussed (Allow to set priorities on podcasts · Issue #2437 · AntennaPod/AntennaPod (github.com)) with having a priority based system even after auto download is reworked.

I’m not sure if we have a ‘Set subscription order’. Could you clarify?

[peakvalleytech]

@RTVance

As a separate comment… please work on this. I am on the 4th time trying to get the downloads that should have been downloaded to download. Because I was having wifi problems I had 200+ files that all had less than 10mb. I had to go in and delete all the files in order to not queue 200 files by hand… needless to say, that would have been my final time using the app.
This is also why I want to option to have the podcasts in the Podcasts or Download folder.

I am currently prototyping some code for this design. I hope to have a workable version on github by the end of October. In the meantime, please keep supporting AP. Your troubles will soon end, my friend!

[RTVance]

@peakvalleytech

I’m not sure if we have a ‘Set subscription order’. Could you clarify?

Setting > User Interface > Set Subscription Order

@peakvalleytech

Yeah, a priority system has been brought up, but it only mitigates the issue. Eventually the same probblems would occur even with priority given a increase in podcasts. That is what a podcasts specific auto download would solve.

I was directly addressing the problem ByteHamster had brought up. When space is limited you need a way to priorities the which episodes to download.

[Matth78]

IMHO if some kind of priority is needed it would be best to have a specific settings for each podcasts. I would suggest a numeric number to offer enough granularity, either 1 to 10 or to accommodate heavy users 1 to 100.
If not set it should be considered as mean between lowest and highest number so that would make it no specific priority. (If same number then subscription order could be used)

Subscriptions order impact display and do not reflect what you consider important. For instance mine is set to alphabetical because I find it more practical.
I would strongly advise to use that.

[peakvalleytech]

@RTVance

Setting > User Interface > Set Subscription Order

Ok, that is what I thought you were refering to. Yes, I hope to include a setting to ‘Sort by priroity’. I alos thought you were refering to a new drag and drop feature i have partially implemented (Github).

I would like to have the priroity be implemented with drag and drop versus a number via podcast specfici setting. I think it is better UX instead of twiddling with numbers. What do you think?

@RTVance

I was directly addressing the problem ByteHamster had brought up. When space is limited you need a way to priorities the which episodes to download.

Right, and feed specific setting should solve this problem. But prioritizing downloads could still be useful.

[peakvalleytech]

@Matth78

IMHO if some kind of priority is needed it would be best to have a specific settings for each podcasts. I would suggest a numeric number to offer enough granularity, either 1 to 10 or to accommodate heavy users 1 to 100.
If not set it should be considered as mean between lowest and highest number so that would make it no specific priority. (If same number then subscription order could be used)

I appreciate your suggestion, but I really don’t think that is the best approach.

There is no way for users to easily compare the priorities of the current podcasts.

With a drag and drop, users can see immediately the relative priorities. And can quickly order the priorities through drag and drop rather than have to individually enter numbers for each podcast. Besides, I have already implemented drag and drop capabilities for the subscriptions screen (#5386, Feed drag drop by peakvalleytech · Pull Request #5386 · AntennaPod/AntennaPod (github.com).

Also once, feed specific auto download is implemented, then extra granulairty won’t be needed as they’re won’t be a fight over which podcasts get to download.
Instead, it provide these two main benefits:

1. ‘it would be nice if these podcast could be download first, so i don’t have to wait’
2. it would provide a way to allow custom sorting, which many user have requested

@Matth78

Subscriptions order impact display and do not reflect what you consider important. For instance mine is set to alphabetical because I find it more practical.
I would strongly advise to use that.

Yes, that could be true. But ‘sort by priority’ would only be an additional option. So users can have the option to change to whatever they prefer.