Can you make this work for other services? about steam-squid

[agentgonzo]

Jarred,

It's not been updated in two years because it still works :-). We use it in our house and it's still going strong.
As it is, it's a Squid proxy, meaning that it will proxy (and cache) any HTTP requests, not just Steam. However, I think that some of the above services use HTTPS to download their content, rather than HTTP meaning that this cache (and others) can't cache the content due to it being a secure connection between your computer and the source.

I'll take a look into and see if any other services download using pure HTTP. If you know/can find out what download site the different games companies use then let me know (running Wireshark whilst updating games is the easy way to find that out)

[jarredwalton]

Hi Gonzo,

I'd love to put together a more comprehensive guide to making this work. I write for PCGamer.com, and last month I went over my 1TB data cap from Comcast. That's likely primarily because I have too many PCs running Steam, and they probably all individually downloaded some large updates. (20GB and 10 PCs can chew through a lot of data!) Besides running this on one desktop, what do I need to do? Looks like I need to set DHCP to point to that desktop as the primary DNS. I'm trying to wrap my head around Docker, squid-in-a-can, and steam-squid all at once, but I'm more of a Windows desktop guy than Windows Server!

Drop me a note if you can:

Thanks!

[agentgonzo]

OK, I've taken a look at it and it seems that valve have changed the URLs that they've been using, which means thath you get a lot more cache misses because the proxy didn't know about the new domains ( valvexyz.steamcontent.com). I've updated the docker image, so if you kill off your existing containers, then run "docker pull agentgonzo/steam-squid" you should see it refresh a new image. Then run it again. It should now work much better with caching the content. I've just tried this on a Windows laptop with docker and can see the inbound network traffic get drop to zero when the proxy is serving cached content. I will get back to you about the -v (volume) settings in a bit, because they're slightly more complicated but that should get you going for the moment. I'm not sure why it's slower when serving it from the disk, but that may be because it's having to make lots of disk saves/reads from a virtual environment - not really sure TBH. If you look at the access logs, you should see that (after you have initially downloaded a game) you should see almost entirely TCP_HIT or TCP_HIT_MEM for steam downloads and virtually no TCP_MISS entries. If you do see TCP_MISS entries after populating the cache, then let me know the URLs that it's you get TCP_MISS for and I'll have another look at it. In short: 1) Kill and delete you existing container "docker kill steam-squid", "docker rm steam-squid" 2) pull newer container (docker pull agentgonzo/steam-squid) 3) Download (small!) game 4) Delete game, 5) Download again - view network activing and access.log for cache hit/miss information. .... unless you got LanCache working in the meantime! Steve

…

On Thu, May 31, 2018 at 6:41 PM jarredwalton ***@***.***> wrote: Hi Gonzo, I'd love to put together a more comprehensive guide to making this work. I write for PCGamer.com, and last month I went over my 1TB data cap from Comcast. That's likely primarily because I have too many PCs running Steam, and they probably all individually downloaded some large updates. (20GB and 10 PCs can chew through a lot of data!) Besides running this on one desktop, what do I need to do? Looks like I need to set DHCP to point to that desktop as the primary DNS. I'm trying to wrap my head around Docker, squid-in-a-can, and steam-squid all at once, but I'm more of a Windows desktop guy than Windows Server! Drop me a note if you can: jarred [at] pcgamer [dot] com Thanks! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABk3H6kaPEHVtnknBTVzAFQH8tr5V-Slks5t4CtOgaJpZM4UUTZC> .

[jarredwalton]

I've almost got Lancache running -- it seems the more robust way of doing things, and I have a NUC box sitting around just asking to be used for such a purpose. But I'm struggling with lancache as well. It's running Debian, and the NUC has an i7-8809G processor with Vega 11 graphics, which don't have X drivers by default. So no GUI. I'm super close to getting lancache working, though, I think! Maybe if I can't do that, I might just set up containers on the NUC and go that route. Ugh... what a mess. I have so much hardware, and this should be easier than it is. LOL

…

On Sat, Jun 2, 2018 at 7:04 AM, agentgonzo ***@***.***> wrote: OK, I've taken a look at it and it seems that valve have changed the URLs that they've been using, which means thath you get a lot more cache misses because the proxy didn't know about the new domains ( valvexyz.steamcontent.com). I've updated the docker image, so if you kill off your existing containers, then run "docker pull agentgonzo/steam-squid" you should see it refresh a new image. Then run it again. It should now work much better with caching the content. I've just tried this on a Windows laptop with docker and can see the inbound network traffic get drop to zero when the proxy is serving cached content. I will get back to you about the -v (volume) settings in a bit, because they're slightly more complicated but that should get you going for the moment. I'm not sure why it's slower when serving it from the disk, but that may be because it's having to make lots of disk saves/reads from a virtual environment - not really sure TBH. If you look at the access logs, you should see that (after you have initially downloaded a game) you should see almost entirely TCP_HIT or TCP_HIT_MEM for steam downloads and virtually no TCP_MISS entries. If you do see TCP_MISS entries after populating the cache, then let me know the URLs that it's you get TCP_MISS for and I'll have another look at it. In short: 1) Kill and delete you existing container "docker kill steam-squid", "docker rm steam-squid" 2) pull newer container (docker pull agentgonzo/steam-squid) 3) Download (small!) game 4) Delete game, 5) Download again - view network activing and access.log for cache hit/miss information. .... unless you got LanCache working in the meantime! Steve On Thu, May 31, 2018 at 6:41 PM jarredwalton ***@***.***> wrote: > Hi Gonzo, > > I'd love to put together a more comprehensive guide to making this work. I > write for PCGamer.com, and last month I went over my 1TB data cap from > Comcast. That's likely primarily because I have too many PCs running Steam, > and they probably all individually downloaded some large updates. (20GB and > 10 PCs can chew through a lot of data!) Besides running this on one > desktop, what do I need to do? Looks like I need to set DHCP to point to > that desktop as the primary DNS. I'm trying to wrap my head around Docker, > squid-in-a-can, and steam-squid all at once, but I'm more of a Windows > desktop guy than Windows Server! > > Drop me a note if you can: jarred [at] pcgamer [dot] com > > Thanks! > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <https://github.com/agentgonzo/steam-squid/issues/ 2#issuecomment-393614628>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ ABk3H6kaPEHVtnknBTVzAFQH8tr5V-Slks5t4CtOgaJpZM4UUTZC> > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AR0nRbObioKlwyulMBAb12feJc5Lv7Svks5t4ptegaJpZM4UUTZC> .