datacamp / courses-introduction-to-shell

Introduction to Shell by Greg Wilson

Introduction to Shell

• Teach: https://www.datacamp.com/teach/repositories/1395
• Campus: https://www.datacamp.com/courses/introduction-to-shell-for-data-science
• Docs: https://instructor-support.datacamp.com

Description

The Unix command line has survived and thrived for almost fifty years because it lets people to do complex things with just a few keystrokes. It helps users combine existing programs in new ways, automate repetitive tasks, and run programs on clusters and clouds that may be halfway around the world. This lesson will introduce its key elements and show you how to use them efficiently.

Learning objectives

• Explain the similarities and differences between the Unix shell and graphical user interfaces.
• Use core Unix commands to create, rename, and remove files and directories.
• Explain what files and directories are.
• Match files and directories to relative and absolute paths.
• Use core data manipulation commands to filter and sort textual data by position and value.
• Find and interpret help.
• Predict the paths matched by wildcards and specify wildcards to match sets of paths.
• Combine programs using pipes to process large data sets.
• Set and use variables to record information.
• Use loops to run the same commands on many different files.

Prerequisites

• None

courses-introduction-to-shell's Issues

ch03ex05 correct answers not accepted

https://campus.datacamp.com/courses/introduction-to-shell-for-data-science/combining-tools?ex=5
grep "2017-06-*" seasonal/spring.csv | wc -l #if they are no other surprises in the data.
grep "2017-06-." seasonal/spring.csv | wc -l could be another answer.

cd allows you to move around the filesystem, not the shell

Is it house style to use a horizontal rule in a multiple choice question?

The example I copied used <hr> to separate description from question in MCQs - is that house style?

include more on grep

from beta: several people asked for more material on grep.

(When) should we introduce csvkit?

csvkit is a set of command-line power tools for working with CSV data, but is not installed by default on any OS.

1. Should we introduce it?
2. If so, should we introduce it in the intro course, or use it as an excuse to introduce package installation in the intermediate course?

ch03ex04 correct answer not accepted

from beta:

https://campus.datacamp.com/courses/introduction-to-shell-for-data-science/combining-tools?ex=4
this is accepted yet only the first item is wanted.
cut -d , -f 2 seasonal/autumn.csv | grep -v "Tooth" | head

add course outline to specs

@gvwilson Ideally we have something like this somewhere in the specs before starting to crank out exercises. This will help us scope the material and make sure we're aligned on course structure (e.g. 4-5 exercises of no more than 16 exercises a piece, including videos).

Support for package installation

Find out whether packages can be installed during lessons so that we can teach brew/apt/yum/etc.

Consensus is that:

1. Packages can be installed.
2. There's no need to worry about persistence across sessions.
3. We do need to worry about intermediate prompts.

Note: we do not need package installation in the basic course (this one), but should introduce it in the second (intermediate) course so that we can show learners what lands where - they'll need to know this in order to debug when things (inevitably) go wrong.

Correct submission not accepted

this is the exercise: https://campus.datacamp.com/courses/introduction-to-shell-for-data-science/manipulating-files-and-directories?ex=10

mv seasonal/summer.csv seasonal/spring.csv backup/ is not accepted but does what is asked.

what's weirder in terms of UX (although it makes sense logically and computationally) is that, after this, the desired solution is accepted, although the shell throws an error:

ch03ex01 correct answer not accepted

Original report from beta:

https://campus.datacamp.com/courses/introduction-to-shell-for-data-science/combining-tools?ex=1
this command is not recoqnized by the shell and an error is thrown
tail -5 seasonal/winter.csv > last.csv

Support for plain-text editor

We need to provide a way for people to edit plain text files, such as their .bash_profile or longer Git commit messages. One option is to try to support something like nano inside the shell emulator; another would be to provide a pop-up dialog whenever the shell wants an editor.

Note: I do not think we need to support vim, emacs, or other full-strength in-terminal editors. In my experience, (almost) everyone who uses those was using them before they came to Software Carpentry, while (almost) everyone who installs an editor to use with the shell installs Sublime, Notepad++, or something else that opens its own window on demand. Having the terminal briefly turn into an editor is pretty confusing for newcomers who are used to each window having a single personality; having a separate widget pop up for editing text will, I think, be less confusing for novices, and completely comprehensible to more experienced learners.

ch02ex05 strange error running head on seasonal

Original comment from beta tester:

https://campus.datacamp.com/courses/introduction-to-shell-for-data-science/manipulating-data?ex=5

I got a very strange error while running head -5 season. I was expecting head: Error reading seasonal

Observed behavior:

1. Error message Cannot open 'season' for reading: no such file or directory
2. Shell session crashes.

Need a "fixme" test as a placeholder

Add some sort of "fixme" test as a placeholder so that we can proceed with the lesson while waiting to figure out how to implement certain things.

Typing '! number' to re-run causes session to hang and requires re-start.

Reported by Rasmus.

Cannot teach multi-line loops until continuation prompt working.

Nick's feedback

General

• Should we let people know about control-C in case they get stuck at the prompt?
• The last chapter gets pretty hard pretty fast. Given this is an intro course, I'm a bit concerned people won't get through it. Let's discuss
• Seems there are several multiple choice questions that could have had a console. Are you sure you chose the right exercise type? (btw, I'm fine with pure MCE, just making sure it's intentional)
• It's nice/helpful if the "success message" is slightly more colorful or informative than "Yes." or "Correct."

Chapters 1-2

• Good if we add some hints (see above)

Chapter 3

https://campus.datacamp.com/courses/intro-to-unix-shell/combining-tools?ex=3

• needs hint - this is hard to do from memory

https://campus.datacamp.com/courses/intro-to-unix-shell/combining-tools?ex=4

• cut -d , -f 2 seasonal/spring.csv | grep -v Tooth worked, but was missing the call to head

https://campus.datacamp.com/courses/intro-to-unix-shell/combining-tools?ex=10

• wc -l seasonal/*.csv | grep -v total | sort -n | head worked, but the call to head was incomplete

Chapter 4

• Good, with hints.

Chapter 5

Feedback relocated to #61.

Add diagram for relative paths

Add diagram to show how relative paths work in chapter 1.

Chapter 5 feedback

Relocated from #57.

Chapter 5

• https://campus.datacamp.com/courses/intro-to-unix-shell/18217?ex=1
  - newbies may not know what IDE means
  - need to give more guidance on what to do after creating the four lines (i.e. ctrl-X, Y, enter). Otherwise lots of people will get stuck here

• https://campus.datacamp.com/courses/intro-to-unix-shell/18217?ex=2
  - UX is a bit weird since you're giving a separate set of instructions in the assignment text and every line prompts an "incorrect" answer
  - could it just be split into two exercises? or just one with more guidance in the instructions
  - GVW: it's already a two-step - discuss?

• https://campus.datacamp.com/courses/intro-to-unix-shell/18217?ex=3
  - hint would be helpful here! -c is hard to remember
  - turns out I was forgetting the period between * and csv :-O
  - would be night to print teeth.out as the final step here to see the results

• https://campus.datacamp.com/courses/intro-to-unix-shell/18217?ex=4
  - this is pretty hard without a hint
  - won't accept this (has to use *.csv)? bash count-records.sh seasonal/* > num-records.out

• https://campus.datacamp.com/courses/intro-to-unix-shell/18217?ex=5
  - isn't this the correct answer? head -n $3 $1 | tail -n $2

• https://campus.datacamp.com/courses/intro-to-unix-shell/18217?ex=6
  - no console? There's no course.txt in the example

• https://campus.datacamp.com/courses/intro-to-unix-shell/18217?ex=7
  - same comment

• https://campus.datacamp.com/courses/intro-to-unix-shell/18217?ex=8
  - typing ls at the prompt caused the exercise to pass!

• https://campus.datacamp.com/courses/intro-to-unix-shell/18217?ex=9
  - what's up with the feedback message?? "required 001..."
  - screenshot here: https://www.dropbox.com/s/28mxkoflvspob5i/Screenshot%202017-10-15%2017.18.43.png?dl=0
  - GVW: that was a leftover debugging message. Should be gone now.

• https://campus.datacamp.com/courses/intro-to-unix-shell/18217?ex=10
  - fix title
  - the hint should probably say more than "you've done this before"
  - I think this would work better as a tabconsoleexercise (vs. bullet) since the tasks are building on each other (and take up quite a bit of real estate on the screen)
  - 4th solution displays incorrectly since it's two lines (need a line break)

no "end of chapter" messages

from beta:

feels weird to end chapters without videos

bug in chapter 3

from beta:

In the exercise: "What's a better way to use one command's output as another command's input?" , the suggested answer is wrong. (answer uses wrong file)

Include one sentence in the course description and first para about what the shell is

Need to say even more clearly up front what the shell is.

exercise templates and more for shell

https://datacamp.slack.com/archives/C23337ZAL/p1502093597727139

Need simple prompt in shell

Shell prompt currently looks like this:

It would be better if it was just a $ prompt (since that's the default for many machines, and requires the least explanation).

chapter 1 8-12, chapters 2-4 10-16

Current totals are 14, 8, 10, and 10.

Go through standard prep prior to beta testing

@gvwilson I'm hoping @sumedh10 can help you with the details here when the content is ready to go. For example, we need to make sure you have a bio, headshot, course badge, etc. cc @yashasroy

Don't use the phrase "duct tape" - it's North American jargon

explain where to find the shell on different platforms

From beta testing:

At the introduction within chapter 1 I missed some information about where to find the shell depending on the OS the user working with.

For example, users being not too familiar with Linux don't necessarily know that they have to open a program called "Terminal" in order to access the shell. The same holds true for Mac OS-users. It isn't mentioned in the course that the shell in Mac OS X can be found under the Terminal App somewhere in the programs folder.

And last but not least I suggest to mention that there's a shell in Windows as well, and that the "cmd" and it's bigger brother (PowerShell) are based on the same idea like a Unix shell but have a different syntax so that you can't transfer the knowledge from this course into a Windows environment.

Remove use of the tree command

Rasmus points out that it isn't installed by default - we can use it to show the filesystem, but shouldn't name it or ask learners to use it.

explain that >> appends to a file

From beta: chapter 3 should explain that >> appends to a file. (Commenter also asked that we explicitly name stdin, stdout, and stderr.)

emphasize flags at the start of the course

from beta:

I suggest that you spend one more word or two about flags / arguments / options. You mention them with the head command for the first time and with the ls command for the second time. However, I think that a newcomer to the unix shell won't understand right away, that he can control nearly each command with a variety of options by notating flags, arguments etc. Somewhat later - with the grep command - this will become clear(er), but my suggestion is to emphasize the power of flags and args right at the beginning of the course.

Add SSH support

Check whether SSH is supported (or can be emulated) in version 1 of the lesson.

The second (intermediate) shell course should introduce SSH: it's essential for accessing clusters and other large-scale computing resources, which is many people's principal motivation for learning shell in the first place.
Note:

1. One way to support it is to spin up two backend processes: one is the user's "host" machine, the other is the (pretend) "remote" machine.
2. We need to limit access so that learners can only reach our fake "server", not random machines elsewhere on the web.
3. We will need to show people how to generate keypairs, but we need to make really sure that they don't use the keypair generated in the exercise anywhere else (e.g., for real work).
4. Before keypairs are introduced, people will be ssh'ing using passwords, so we need to handle intermediate prompts (discussed in a separate issue).

Feedback on course specs/dev process

@hugobowne commented on Thu Aug 10 2017

This is feedback on @gvwilson's doc on course specs & development process as of this commit;

cc @ncarchedi as he'll be interested.

General

This provoked a great deal of thought on my part and I think the potential shiftings of process and specs are exciting.

• Something to note is that the process proposed doesn't resemble our current process much at all; this is fine, but worth thinking about as it seems to be more a new way of doing course development here at DC, rather than a re-working (to be clear, I'm not against this);
• These specs and process seem, to be honest, intuitive, natural and great for both instructors and students; I presume they also arise from years and years of Greg thinking about, reading literature on and practicing and honing such curriculum development techniques, which means they will be inevitably data-driven in more than one way; however, to play devil's advocate, if we're to make radical changes to the current specs/process, it would be good to see data or findings suggesting that they work; Can you point us to such resources? Perhaps this? Or any of Mark Guzdial's work?

I'll now provide feedback on each section. It's also tempting to provide feedback on the embedded specs for the unix course but I'll try to hold off.

Step 0: Learner profiles

• This is fantastic; the current approach to letting an instructor know who the students 'generally are' is skills-based and relatively ad hoc; in addition, I've had numerous instructors say, at various points in development, when I've said 'such and such material may be too advanced', for example, that they don't think they quite get who we're aiming the material at;
• Any specific hard-skill profile info.? e.g. 'knows what a group-by is'? I know that this is pretty fine-grained but it is a challenge we come up against all the time; thinking about re-structuring additional resources could help, e.g. doesn't matter if a given student doesn't know group-by, we can send them to a couple of exercises somewhere or a blog post

Step 1: Concept Map

I know I said I wouldn't provide feedback on the unix stuff but this unix concept map is a thing of beauty.

• This is great and useful for curriculum dev, thinking out most productive methods will be key, e.g. flow charts or directed acyclic graphs
• I agree mostly w/ resisting the assumption to writing out learning objectives at this point, however:
  • I think instructors and CLs will already be developing ideas of learning objectives in their heads while engaged in the process and, when multiple people are doing so, it may be worth jotting a few things down to make sure that we're not thinking at cross purposes (may be mitigated by concept map); i'm not suggesting anything extensive and set-in-stone, but rather some general notes;
  • learning objective have been a good way to communicate to the rest of the company and externally what courses contain or will contain; could a concept map take this place?

Step 2: Summative Assessment

• This is key and perhaps takes the role (and performs it better) of learning objectives.
• The focus on practical and exemplary examples of what the student will be able to do will be valuable for instructors, students, marketing and many more stake-holders

Step 3: Formative Assessment

• This is all reasonable so not so much critical feedback
• Only thing I'd point out is that, in the way the dev process currently works, I think this would be owned by Curriculum Lead as it will form part of the functional specs

Step 4: Sequencing

• the same feedback as Step 4

Step 5: Chapters

This is also the point at which it's worth writing formal learning objectives. Doing it earlier often wastes effort, since material may be cut in Step 4.

• As stated above, while doing all the above steps, it's not much wasted effort to jot down several notes to make sure that everybody is aligned (nothing formal); thoughts?

Step 6: Course Description

Brilliant that this now flows directly out of the work that's already been done. The unix course description is likely one of the best I've read: how did you write it before all the other steps were complete! ;)

ch03 generalize SCT

from beta:

one question requires cut -f 2 -d, seasonal/summer.csv | grep Tooth -v but will not accept cut -f 2 -d, seasonal/summer.csv | grep Tooth -v (only differences is in spacing) although the code produces exactly the same out put for both.

ch01ex04 correct answer already selected when exercise starts

from beta:

https://campus.datacamp.com/courses/introduction-to-shell-for-data-science/manipulating-files-and-directories?ex=4
the correct answer was already selected when i moved to the exercise.

Support for continuation prompts

The first shell lesson (this one) can get by w/out support for continuation prompts (e.g., those produced by pagers like more and less, or the password prompt produced by sudo), but the second (intermediate) lesson will need them. We will also need support for ^C, and it would be very useful to have support for ^Z so that we can teach process control.

Tell people there need to be spaces between commands and arguments

ch02ex10 typo first paragraph

https://campus.datacamp.com/courses/introduction-to-shell-for-data-science/manipulating-data?ex=10
typo first paragraph.

Teaching multi-line interactive loops

Typing paste on its own without arguments produces a timeout message

1. In a ConsoleExercise, type paste without arguments, switch focus, come back: a timeout error message is displayed. (Switching focus may not be necessary.)
2. Type paste on its own on a Mac OS X command line - it produces a one-line summary of behavior.

The latter is preferred.

Incorrect submission thrown for exploration

If this has already been discussed, feel free to close.

In many exercises, I want to see where I am (pwd) or explore a little bit (ls). Sometime it is helpful to see where I am to solve the challenge, such as here: https://campus.datacamp.com/courses/introduction-to-shell-for-data-science/manipulating-files-and-directories?ex=11

however, when I execute pwd, I am told I am making an incorrect submission. But I'm not even trying to make a submission. This is proving very frustrating for me. However, it may be a result of expectations set from Python/R courses on DC.

Need a way to re-set the filesystem to its starting state.

Some of the exercises modify the filesystem; need a way to re-set while inside an exercise. For example, if the learner types the wrong thing for the "Deleting files" exercise in chapter 1, Rasmus noticed that re-tries are being done in a mangled filesystem.

1. Tell learner to refresh before resubmitting (possibly in the error message)?
2. Support a per-exercise flag to tell the system to refresh on each try?

Course doesn't build

@ramnathv We're trying to get the course to build on teach but can't figure it out. We've mimicked what's in the test course repo. Please let us know the best way forward. cc @gvwilson

Add notes on naming files

http://www2.stat.duke.edu/~rcs46/lectures_2015/01-markdown-git/slides/naming-slides/naming-slides.pdf

ch02ex07 correct answer not accepted

from beta

https://campus.datacamp.com/courses/introduction-to-shell-for-data-science/manipulating-data?ex=7
The session take time to load
the answer is not accepted.

Direct instructions in course specs at instructors

Currently, it feels like the instructions in the course specs are directed at internal team members (CLs + CDs), but they should really be directed at external instructors. Given that this part of the process will be "owned" entirely by the CL, it's probably not important to make the distinction between CL and CD in the specs. It's the CL's responsibility to set the instructor's expectations around the review process and handoff.

Make `man` available in the shell.

Create and install author and shield image files

ch01ex05 enforce use of absolute paths

From beta:

https://campus.datacamp.com/courses/introduction-to-shell-for-data-science/manipulating-files-and-directories?ex=5
in this exercise, i was using mostly the relative path and all answers were accepted. I
think to test the students even further. I think you should make defining the absolute
path mandatory in one task. To test understanding.

Introduce ctrl-C

Need to find a natural place to introduce ^C to kill running job.

Set HOME to /home/repl

Must change HOME to /home/reply for exercises in chapter 5.