Scheduled Direct duplicate program requests about sagetv

[JustFred51]

What's the expectation for how often these files would be created/updated? Will this feature only be enabled if specifically selected? I'm wondering about the impact to the life of an SSD if there are a significant number of writes.

[Narflex]

How did you come across this? What's the current negative impact somebody can experience from this?

[enternoescape]

The files are relatively static. There are a lot of them, but they are very small and the churn should be very low. If that's a bigger concern over disk space used, we could make cleanup only happen when files are older than 30 days.

[enternoescape]

@Narflex Bob from Schedules Direct is re-writing the backend in Python and they have a mechanism that attempts to thwart DDOSing. This is one of the things it may temporarily block traffic from a specific IP over. It came up while I was testing several weeks ago and I've been trying to find time to work on a solution.

[Narflex]

OK, I see how that would be a problem then. So what are the specific cases we have right now where we are re-downloading the same data? This may be better solved by adding another field to some DB objects to track this instead.

[enternoescape]

I need to do a little more to find the worst offending lines of code, but the issues are around recommendations and getting series descriptions and images. I'll try to hone in on what specifically we're gathering (it's been a while since I've looked at some of this code), but I still think the extra data would only slow down startup and eat up memory for content that doesn't need to be available for any time other than when we update the EPG.

[enternoescape]

I've boiled it down to two options because I don't want to leave this issue unchecked and I really do not have much personal time to write anything significant. Either we can use the caching I proposed or I will just remove all the code that references programs outside of what is needed to fill the EPG. That may mean we get less series data or no series data and that could result in less images to display in the UI.

[Narflex]

What is this you are referring to? "I will just remove all the code that references programs outside of what is needed to fill the EPG."

[enternoescape]

I would remove API calls to the Schedule Direct programs API that are only an attempt to fill in missing series data. I know you're just trying to better understand my intent, I can assure you the end result would be useable, just less images. The alternative is that I fix everything over the span of maybe a year and in the mean time accounts get blocked repeatedly.

[Narflex]

I'm fine with you doing it the way that gets done sooner; just wanting to understand the implications more. Like the celebrity images, it seems like only trying to generate series info for ones that don't exist would greatly reduce the number of requests:

sagetv/java/sage/epg/sd/SDRipper.java

Line 1764 in aff35f9

// Add/update the series info any time an episode of a series is updated. This

That seems to indicate we try to update it for every show every time...so if you reduce that to only get ones that don't exist in the DB that should be fine; and maybe that is what you are actually suggesting. :)

[Narflex]

@enternoescape If you just want to go ahead with the caching solution you originally proposed I'm fine with that too. :)

[enternoescape]

@Narflex Sorry, I've been getting pulled a few directions over the past few weeks. I will try to get something submitted before July.

[Narflex]

No problem at all...I really appreciate the help here so whenever you get it done is fine. I just wanted you to know I'm not trying to push you towards any specific solution. :)