joergi / magpidownloader

Downloads all the Mag Pi (Raspberry Pi Magazine) on your Windows, Mac OS or Linux or with Docker

MagPiDownloader for Windows, Mac, Linux and Docker

All the issues are downloadable for free on https://www.raspberrypi.org/magpi/issues/
You can also buy all the paper versions of this issues on http://swag.raspberrypi.org/collections/magpi

This project should save you time, and let you download all scripts together.

Linux and Mac user: here is your tutorial

Windows user: here is your tuturial

Docker - needs to be fixed - issue [89, 26, 25, 42]

---

This project was done by this lovely people, thank you!!!

magpidownloader's Issues

fix regular issues on powershell

at the moment there is an error in the testing (also in the running?) of the download of the regular issues on Windows Powershell

as I'm not having a Windows machine to really debug it locally.....

Test with errors: https://github.com/joergi/MagPiDownloader/actions/runs/3253313845/jobs/5340465575

Auto build Docker file

run an automatic build for every update

use generic downloader (Mac/Linux)

extract downloader for:

• MagPi
• HelloWorld
• Wireframe
• Hackspace Magazine
  into an own project

Update latest issue automatically

use the https://github.com/joergi/MagPiDownloader/blob/main/linux_mac/magpi-issue-downloader-latest.sh from @keni-n to automatically update every day the latest issues via Github Actions

Using generic downloader script for Windows

same as #39 but for Windows

• use it for books
• use it for regular issues

adapt the generic file for the docker script

Dockerize it

making a dockerized version and push it to docker hub.

fix docker build

fix the docker build, so it's working as it was working before the code changes

Windows: .\magpi-special-issue-downloader.ps1 is not working

Beim .\magpi-special-issue-downloader.ps1 gibt es Fehler:

PS E:\MagPi\MagPiDownloader\windows> .\magpi-special-issue-downloader.ps1
Downloading Essentials_Games_v1.pdf
Ocorred an error trying download  +
Downloading Projects_Book_v1.pdf
Ocorred an error trying download  +
Downloading Projects_Book_v2.pdf
Ocorred an error trying download  +
Downloading Projects_Book_v3.pdf
Ocorred an error trying download  +
Downloading Essentials_Bash_v1.pdf
Ocorred an error trying download  +
Downloading Essentials_SenseHAT_v1.pdf
Ocorred an error trying download  +
Downloading Essentials_Sonic_Pi-v1.pdf
Ocorred an error trying download  +
Downloading Essentials_Scratch_v1.pdf
Ocorred an error trying download  +
Downloading Essentials_Minecraft_v1.pdf
Ocorred an error trying download  +
Downloading Essentials_GPIOZero_v1.pdf
Ocorred an error trying download  +
Downloading Essentials_C_v1.pdf
Ocorred an error trying download  +
Downloading Essentials_Camera_v1.pdf
Ocorred an error trying download  +
Downloading Essentials_AIY_Projects_Voice_v1.pdf
Ocorred an error trying download  +
Downloading Beginners_Book_v1.pdf
Ocorred an error trying download  +
Downloading Annual2018.pdf
Ocorred an error trying download  +

Die anderen Skripte klapten.

add Windows testing to travis-ci.yml

The Windows Powershell scripts needs to be tested with travis too.

Docker: last file is not downloaded (at least in the tests)

Somehow the last issue is not downloaded:

https://github.com/joergi/MagPiDownloader/actions/runs/8224658977/job/22488748591?pr=223

docker run  --privileged -u root -v $(pwd)/issues:/app/issues/ joergi/mag-pi-downloader -f 137

only downloads 137 but not 138

Fixing the last linter issues Dockerfile (windows is solved)

I fixed a lot of the the linter issues.

but the last ones I have no idea how to catch them:

for the windows file
https://github.com/joergi/MagPiDownloader/runs/1202964598?check_suite_focus=true#step:4:59

for the Dockerfile:
https://github.com/joergi/MagPiDownloader/runs/1216756825?check_suite_focus=true#step:4:172

It would be super great, if someone can help here.
even if you can just fix one of this 2 issues... THANKS

test if downloaded file exists in testing.yml

as done here: joergi/HelloWorldDownloader#6

All special edition downloaders should be using the same file for all OS

At the moment the Linux and the Windows downloaders don't have the sources duplicated.
For each special downloader should be a file which only contains the links, and the downloaders should deal with it

For example:
Windows for Essentials_Games_v1: https://github.com/joergi/MagPiDownloader/blob/master/windows/magpi-special-issue-downloader.ps1#L21
Linux for Essentials_Games_v1: https://github.com/joergi/MagPiDownloader/blob/master/linux_mac/magpi-special-issue-downloader.sh#L21

keep it simple, one file to serve both downloaders.

Similar to joergi/HackspaceMagazineDownloader#5

latest issue should only be raised, if download works

At the moment the latest issue is triggering every time a new pull request.
But the download is only available after 20 days after release (or something like that)
so the download-latest script should check if the download should work already or not!

[Windows] - Powershell script should use issues.txt too

The Windows powershell script should also use the issues.txt file as the Linux one.

fix windows download for special issues

fix Magpi-special editions

somehow the download of special editions are not working
if you go the url https://magpi.raspberrypi.org/books/essentials-bash-vol1/pdf the website wants you first so spend some money, or you can click to direct download.

somehow the code of the downloader needs to be changed to accept these links too.

 printf -v page_url $downloadUrl "$i"
		pdf_url=`curl -sf $page_url | grep "c-link\"" | sed 's/^.*href=\"//' | sed 's/\(\?.*\)\?">.*$//' | sed "s#^\(/.*\)#$siteUrl\1#"`
		wget -N $pdf_url -P $outputDir
		i=$(( i+1 ))

Invoke-WebRequest not found

Anyone that is using an older version of powershell will not be able to run the scripts by default because Invoke-WebRequest was not available before PowerShell v3.0.

One needs to update PowerShell to at least v3 in order to use this repository in the default state, or changes need to be made to the ps1 scripts to allow it to be backward compatible with PowerShell < v3.0.

The exact error that is thrown is below:

The term 'Invoke-WebRequest' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At D:\Git\MagPiDownloader\windows\magpi-issue-downloader.ps1:34 char:28
+     $l = (Invoke-WebRequest <<<<  -Uri $baseUrl -usebasicparsing).Links
    + CategoryInfo          : ObjectNotFound: (Invoke-WebRequest:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Sync only new issues and remove dependency on hardcoded issue count with wget mirror

First off; great idea! Now you've got me wondering if I can run this directly on my hacked E-reader.

Forgive me if there's a reason I've missed for your method, but wget actually has some super powerful features for accomplishing this.

Here's an example for English MagPi issues using the long-winded wget option names so you can see what's going on:

wget --recursive --no-directories --no-parent https://www.raspberrypi.org/magpi-issues/ --directory-prefix issues --mirror --accept-regex "/MagPi[0-9]+\.pdf"

This will download all issues in the format MagPi00.pdf into the issues folder. It will skip an issue if it's already downloaded.

test the dockerized build

test the dockerized build

make docker also working for the special issues

at the moment the docker downloads are only working for regular issues

change the script or make a new one, so it also works for the books

URLs for downloads have changed, scripts no longer work

For issue 89 for example:

https://magpi.raspberrypi.org/issues/89/pdf

Thus, the scripts need updating.

Jake

Docker should use a remote source file

At the moment, after each update of the magazine issues source file, docker must be rebuild, to have the newest issue.

This can be changed if (at least) docker uses the remote file.

Maybe a

docker=true ./magpie....sh

Should be hopefully enough

Availibility to set the name for downloading

same as

making the shell scripts usable for windows powershell

At this time, the scripts are only working on Linux and OSX machines.

With Windows 8 and 10, Microsoft has introduced the Powershell.

So there is a good opportunity to bring the MagPiDownloader to the Windows system too.

I started to do this by myself, but it was not that easy.
I found on StackOverflow, that it should work like this:

$client = new-object System.Net.WebClient
$client.DownloadFile("https://www.raspberrypi.org/magpi-issues/Essentials_Games_v1.pdf","special_issues/")

But maybe there was something wrong, or I started it wrong.

That's why I ask here for help.
Thanks a lot.

Getting the Windows scripts working with specific range

On Linux you can download a specific range of normal issues like this bash magpi-issue-downloader.sh -f 42 -l 52
but on Windows it's not running yet with: Download a specific range of normal issues: ./magpi-issue-downloader.ps1 -f 42 -l 52

The script maybe needs some enhancement, to get it running with the specific range

using issue number from file for automated testing with latest issue

use output issue.txt for testing always with the latest version.

    - name: Download Issues with start and end limit
        id: normalIssuesWithRange
        run: bash ./linux_mac/magpi-issue-downloader.sh -f 94 -l 95

should become something like:

    - name: Download Issues with start and end limit
        id: normalIssuesWithRange
        run: bash ./linux_mac/magpi-issue-downloader.sh -f ($recent_issue - 1) -l $recent_issue

The normal recent_issue comes from this file, https://github.com/joergi/MagPiDownloader/blob/main/sources-for-download/regular-issues.txt

fix Linter for docker

at the moment the docker linter is disabled, as it would make error.

one possible solution:
make the test files into one for each system. but that's something I want to avoid.

[QUESTION] - It should only follow, if a redirect happens

hey @keni-n - I need your help for minute.

you introduced the code:

		printf -v page_url "https://magpi.raspberrypi.org/issues/%02d/pdf" $i
                pdf_url=`curl -sf $page_url | grep c-link | sed 's/^.*href=\"//' | sed 's/\?.*$//'`
		wget -N $pdf_url -P $OUTDIR

I understand, that (for the magpi links) it will follow in the 2nd line the redirect and download the new stuff.

In theory, If I want to use that for links, which are not forwarded, I can of course use the wget directly, as done before.
can I make somehow line 2 something like:

if follow 
                pdf_url=`curl -sf $page_url | grep c-link | sed 's/^.*href=\"//' | sed 's/\?.*$//'`
else
                pdf_url=`curl -sf $page_url`
/if

of course this is bullshit code, but can I write this ?
Thanks a lot

Picture of a staple of real MagPi paper issues or create a nice design / logo

At the moment, if I share it on twitter, it always shows the picture of me, the maintainer.

I really want to replace it with a picture of a staple of real MagPi paper issues... the higher the better.
Or you can design a nice logo for the MagPi Downloader instead?

The picture / design must be committed under the same license as the code. If you commit the picture, you will promise, that you have the right to commit it under GPL-2 License so I can use it in this project.