View Users Solutions about open-api

[utsab]

Before we make any decisions on this, I'd like to do a few user interviews with teachers to get a better sense of how much they really want to see students' solutions. My intuition is that they would more highly value statistics like how many attempts a student has made or how much time spent on a challenge, not necessarily the content of the solution.

Would it be possible for the api to return the number of attempts on a given challenge, or how much time a student spent on it? If so, that would be fantastic.

I plan on chatting with a few teachers this month to validate my assumptions here.

[imcodingideas]

@ojongerius I would be totally OK with something like that. Another solution would be to have a lamda function that deletes the solutions for Users who are inactive for some number of period. Say, 6 months.

This way the database size, and cost does get reduced and maintainable. I do have experience with AWS and would love to provide development support and financial support.

[QuincyLarson]

Hi everyone, I just got back from China so pardon me being a bit late to this discussion.

For the sake of 100% clarity, and to summarize this discussion so far:

1. we plan to continue storing completed algorithm challenge solutions and the submitted URLs for projects
2. the rest of the challenges are designed to be as bite-sized as possible (they can usually be completed in less than 2 minutes). Yet the solutions for these challenges are taking up ~75% of the total volume of our database, and making our user records extremely large and our entire platform slow, so we don't plan to store them any more.

I recommend we use the forthcoming freeCodeCamp open API as an opportunity to build a separate client for classroom mode that stores not only freeCodeCamp's normal data, but also any other information a teacher might want to collect, such as the some of the attributes mentioned in this thread, like how long the lesson took, previous solutions, etc. This would allow Classroom Mode to eventually offer a much broader range of customizations options without having to wait on the core learning platform.

Eventually Classroom Mode may turn into its own self-hosted application. I could envision schools, universities, and even governments running their own instances of this, and they would want as much control over their data as possible. By collecting as little data as necessary, freeCodeCamp can both honor our users' right to privacy, while also giving heavily regulated organizations maximum flexibility in how they use freeCodeCamp.

Regarding donations, I would encourage you instead to put money toward hosting an example classroom mode sandbox environment and a support forum for teachers and administrators who want to adopt freeCodeCamp at an institutional level.

[utsab]

Hi everyone,

We have performed 3 user interviews with teachers at CSUMB, CUNY, and St. Francis Highschool. Here are the main takeaways:

• We confirmed that they do not care much for the individual solutions to the drills. We can safely remove this feature from classroom mode.
• They do care a lot about two key statistics: 1) number of attempts per challenge 2) time spent on each challenge. Higher priority to number of attempts.
• Side note: The student profile page on learn.freecodecamp which lists all individual challenges would really benefit from a redesign in order to give more insight to teachers. I can recommend someone who would be happy to do this work if you approve of the redesign.

Request: Can you collect number of attempts per challenge? This will seriously help the MVP for classroom mode.

[ojongerius]

I think solutions will no longer be stored at freeCodeCamp. @freeCodeCamp/open-api can you confirm this?

[raisedadead]

That is the plan. We are removing all solutions other than the links to user submitted projects. I understand that its a breaking change for class room mode.

But I think @imcodingideas you should probably connect with @QuincyLarson for more confirmation.

/cc @Bouncey

[user512]

It's a surprise to me. Let's say a student is only half way thru a challenge in FCC, and the student decided to complete the challenge tomorrow, the student will have to start from the beginning?
It sounds like fcc still have to store student solution somewhere.

[Bouncey]

We are still storing solutions in localStorage, which are then removed on successful submission. So half completed solutions, that have had the tests ran on, them will still be held in that browser which is the case in production right now.

Currently, freeCodeCamp only stores completed solutions in the database.

This is a major contributing factor to the size of our user records and, in some cases, the slowness of queries against these records. Currently our user records alone are touching 70GB and will only get bigger. After a planned migration coming soon, which includes the removal of non-project solutions, our entire user data set will be around 10.5GB.

[user512]

Thanks @Bouncey for the explanation, this is a smart way to operate without compromising user experience.

[imcodingideas]

@Bouncey isn't that the benefit of graphql, so we're able to query the data that we need.

[Bouncey]

@imcodingideas 100% it is. But with a larger data structure we have a higher overhead for controlling user privacy and storage/transfer costs of backups and production database instances.

We are making an effort to reduce db costs for challenges so we can expand the value of future services provided by freeCodeCamp, as always, for free. This will become more evident in the coming months, and will lean on a new graphql api supported by this project.

Being 100% donor supported, we have to watch those pennies.

[imcodingideas]

As I see this data, defining the success of classroom-mode - what if I were to donate a significant part of last years backup cost. Would that enable us to at least keep the data for now? @QuincyLarson please connect with @utsab for my contact information.

[ojongerius]

Could a feasible solution be to store solutions for students if, and only if they are registered with a classroom?

I know this is currently not implemented. We'd have to provide support for classrooms in open-api. However with the classroom team providing development support, and possibly even financial support, why not?

[ojongerius]

Great!

Let's see what @utsab comes back with first.

[QuincyLarson]

Hi @utsab Interesting. Thanks for sharing these takeaways!

They do care a lot about two key statistics: 1) number of attempts per challenge 2) time spent on each challenge. Higher priority to number of attempts.

We thought about tracking number of attempts in the past but our reasoning has been that it might be discouraging. We track the average time on each challenge through Google Analytics, and it's generally in the 2 - 4 minute range. When we notice challenges that take longer than that, they're good candidates for rewriting or breaking up into smaller challenges.

Side note: The student profile page on learn.freecodecamp which lists all individual challenges would really benefit from a redesign in order to give more insight to teachers. I can recommend someone who would be happy to do this work if you approve of the redesign.

We would welcome suggestions for the code portfolio.

Our current goal regarding storing solutions is to figure out a way to store them in S3. They are too large to store in a database at the scale we're operating at (our database would grow at the rate of a gigabyte per day!) We may get to this in August or September. Other functionality should be available through the Open API before then, though.

[utsab]

HI @QuincyLarson, thanks for considering our requests. I think there may be some miscommunication regarding the number-of-attempts metric. You wrote this:

We thought about tracking number of attempts in the past but our reasoning has been that it might be discouraging.

Yes, I agree this would be discouraging to show the students. However, the intention was to show this to the teachers not the students. The use case here was that a teacher could see, for example, that one topic had 10 times the number of attempts than the other challenges. The teacher could then conclude that students needed more reinforcement on that topic and then devote extra class time to review it.

Does this make sense? If you agree with this, we would really love to have that number-of-attempts metric exposed in the api. Can we have it Quincy?

[QuincyLarson]

@utsab I just talked with another organization yesterday who wanted this sort of information as well. I think it's fine for us to store it as long as we carefully control access to it.

We could store the number of attempts, completion time, and solutions, and then only expose these stats to the individual users and other users whom they have verified as their teacher.

[utsab]

@QuincyLarson I'm glad you're open to storing this information. I hear your concern about carefully controlling access to it. One idea we were considering was to allow campers to enter a "class code". Only the teacher with the correct classcode would have access to the data.

Question: I'd like to volunteer to implement the appropriate features:

1. Storing the number-of-attempts in the learn.freecodecamp database
2. Exposing the field in the open-api
3. Implementing the class-code feature in learn.freecodecamp to control access.

Would it be alright if I submitted pull requests for these features, or do you think these tasks are best left for more senior folks on your teams? I would love to work on it personally if only to speed up development of classroom-mode.

[imcodingideas]

I'm thinking maybe @ojongerius, @raisedadead, or @Bouncey maybe could create a ticket with the things we would need to complete as more of a direction/leadership.

[utsab]

@QuincyLarson, I submitted a feature proposal for the class code. I believe this will address the concern you have around protecting access to the number-of-attempts data.

I look forward to hearing any feedback you have on this.