setting `--commitGenerated` to false causes npx projen to fail when ran in a pipeline or cloned repository. about projen

[mrgrain]

I am not sure what you mean by required, is it similar to how the .gitignore is required and still committed even if the setting is set to false?

I mean setting the commited option to true for that file.

[rdewaele]

For future reference, the commit about commitGenerated is here (I think 😅): #1972

[mrgrain]

The digression

(1) We need to commit package.json because otherwise you can't install.

I presume npx projen assumes (reasonably so) its dependencies are installed?

Would it be technically possible to restore a projen project starting from only a .projenrc.ts file? (I'm asking this to validate my assumptions, I understand that actually supporting this is beyond practical.)

Here's some background knowledge for you: The projen cli is pretty dumb and mostly a wrapper around .projen/task.json. If you go and find the default task in it, you'll get this: ts-node --project tsconfig.dev.json .projenrc.ts. Or if we want to reduce this to a simpler form for the sake of the discussion: node .projenrc.js. At this point you might have realized that synthesizing a projen project, is nothing but running a Node.js program. And Node.js programs cannot run unless all its dependencies are available.

We could imagine a simple case where the projen program only depends on the projen package. In that scenario, it would be possible to install the missing package and then run the program. Or maybe we could even find a way to get the dependencies from the invoked cli, since at least for now these are the same package. In fact, the projen new command is a more generalized version of this: It will install projen plus up to one other dependency via --from into a directory, create a projen program in memory and execute it.

But unfortunately this all falls apart as soon as be have created the project for the first time, because we have no guarantees anymore which dependencies the program requires. So I think the answer has to be: No, in the general case it's not technically possible to restore a projen program from only a projenrc file.

As a further fun though experiment, we could look at other languages or ecosystems. Particular Deno comes to mind. For some reasons you can't use projen with Deno at the moment. But if you could, it would be possible to restore the full projen project from just the projenrc file, because Deno stores dependency inside code and can install any missing deps on the fly.

[mrgrain]

Thanks! yes that's a fun edge case. I'm inclined to say that the tsconfig.dev.json should be marked as required in this case. What do you think?

[collin-buckens]

I am not sure what you mean by required, is it similar to how the .gitignore is required and still committed even if the setting is set to false?

I don't have a strong opinion regarding the solution for this problem because I have very little experience with typescript development.

[rdewaele]

I had a discussion recently with a colleague about this, and our conclusion boiled down to:
"You need to run something on a fresh clone anyway, even if only npm install, so why commit files that are generated at all? (1)"
I.e. Just run projen, which generates all the things and init tasks and everything.

(1) Of course, when commitGenerated is false - no intent to start a discussion whether committing generated files or not is the better alternative. I suspect that discussion is one of the reasons this option exists in the first place. 😅

For me, this begs a few similar questions:

• Why are files still committed when commitGenerated is false at all?
• How does the projen project decide what should be marked as required?
• Can projen even arbitrate in the general case on what is required and what isn't?
• Should this be configurable as a list of what is required?

Maybe all questions can be answered with the rationale for the existence of the commitGenerated option? 🤔

[mrgrain]

@rdewaele this is an absolutely valid viewpoint and why commitGenerated in l exists in the first place. You can try and find the PR introducing it for an extensive discussion and tbh some should write a documentation page about both options.

The reason why some files need to be committed is purely functional. Examples:

(1) We need to commit package.json because otherwise you can't install.
(2) We need to commit .gitignore because it needs to exist to prevent committing files.

This where I'm coming from in regards to the tsconfig that is required to run projen default. You need it to run it.

---

You might want to argue that falling back up a default TS config would be perfectly fine. My counter arguments to this is that using a default config as a fallback only will introduce inconsistency between different runs of projen default. Also, the check if the file exists or not is not super straightforward and needs to be done in shell command.

My suggestion here for a full solution is to do these two:

• Mark the tsconfig as committed if it's configured to be used by the projen default command. This is consistent with the current definition of the feature.
• Introduce a new option to have projen default run without a tsconfig (e.g. the JS version doesn't need one). This can be just the default config, or anything that can be committed as cli option to ts-node

[rdewaele]

The first part is a digression on the why of the option, the second part is pertinent to the concrete question at hand.

---

(1) We need to commit package.json because otherwise you can't install.

I presume npx projen assumes (reasonably so) its dependencies are installed?

Would it be technically possible to restore a projen project starting from only a .projenrc.ts file? (I'm asking this to validate my assumptions, I understand that actually supporting this is beyond practical.)

(2) We need to commit .gitignore because it needs to exist to prevent committing files.

Well, it can be generated by running npx projen before staging files. Are practical considerations involved in this reasoning? 🤔

I did not find an explicit rationale by reading #1972, other than for GitHub flows. The stated goal is just to have cleaner repositories while keeping the same developer experience.

So I am still left with the question as to how exactly it is decided whether a file should be committed even though it's generated, and commitGenerated is set to false.
I apologize if this is a misunderstanding on my end, but the current train of thought seems to be commitGenerated=false => "do not commit anything ... unless it results in something not working then commit it anyway"?
This would be consistent with "keeping the same developer experience", but not quite rigorous.

I think I can find an answer for myself in the case of GitHub flows: having projen define the flow would require a flow to run projen to create a flow. Assuming this could work in the first place, it's an indirection that doesn't bring any value, and there's still a flow committed anyway.

For e.g. .gitignore, I'm currently unable to find a similar explanation. Not committing it would result in a different developer experience though.

On the other hand, if the goal is to not commit generated files iff there is no observable difference in developer experience, why not have an opinionated choice in the first place and not commit them? It's bad form to edit them anyway. 🤔

---

More directly on topic, if marking tsconfig.dev.json as required (for TS projects) is consistent with why other files were marked as such, I think it's a good solution.

My counter arguments to this is that using a default config as a fallback only will introduce inconsistency between different runs of projen default.

Technically one could generate until the file no longer changes (fixed point), but committing the actual file seems more sane.

Also, the check if the file exists or not is not super straightforward and needs to be done in shell command.

Framing it this way, checking whether the file exists starts feeling like a workaround for a wrong decision earlier in the chain.

[mrgrain]

The Concrete

On the other hand, if the goal is to not commit generated files iff there is no observable difference in developer experience, why not have an opinionated choice in the first place and not commit them? It's bad form to edit them anyway. 🤔

Projen has made the opinionated choice to commit all files. This is the baseline. We can obviously discuss the advantages and disadvantages of this, but it seems like we agree that both are valid choice. I believe it's very unlikely that we are going to change the default for this.

So I am still left with the question as to how exactly it is decided whether a file should be committed even though it's generated, and commitGenerated is set to false. I apologize if this is a misunderstanding on my end, but the current train of thought seems to be commitGenerated=false => "do not commit anything ... unless it results in something not working then commit it anyway"? This would be consistent with "keeping the same developer experience", but not quite rigorous.

You are probably looking for a complete definition that doesn't exists. If we loosely apply Occam's razor, the simplest explanation is that the author of #1972 prefers the clean option and what we have today is the result of whatever the PR author and reviewer came up with. I'd love for someone to take a stab at defining this better and then also document it somewhere for posterity.

I very much agree with you that all attempts of a definition so far are very vague. I don't like "keeping the same developer experience", because that feels very subjective. "Unless it results in something not working" is a big improvement over it. However "not working" still allows for a very broad range of interpretations; from "it must (figuratively) explode" to "not working as expected".

I did not find an explicit rationale by reading #1972, other than for GitHub flows. The stated goal is just to have cleaner repositories while keeping the same developer experience.

Which is curious. With the risk of over interpretation, thus might indicate that for the original PR author & reviewer marking .gitignore as needs-to-be-committed was the obvious choice.

I think I can find an answer for myself in the case of GitHub flows: having projen define the flow would require a flow to run projen to create a flow. Assuming this could work in the first place, it's an indirection that doesn't bring any value, and there's still a flow committed anyway.

I think it's flat-out not possible to generate this dynamically. Workflows have to exists in the repo to be executed.
A similar easy case is .gitattributes. GitHub is using this file to make some UI decisions, so it has to be committed.

(2) We need to commit .gitignore because it needs to exist to prevent committing files.

Well, it can be generated by running npx projen before staging files. Are practical considerations involved in this reasoning? 🤔

For e.g. .gitignore, I'm currently unable to find a similar explanation. Not committing it would result in a different developer experience though.

It definitely is a practical consideration. And to me it reminds me of the straw man discussion whether we even need a .gitignore file. After all, if you don't want a file in the repo just don't commit it! ;) Let's look at the most basic workflows, assuming commitGenerated=false and that it would include .gitignore:

The dev clones the repo locally. To do anything, they have been told to run npm install first. They do that.
Now straight out of the starting block they will get a ton of files in node_module in their git diff. Only once they have run projen default, these files will disappear.

And this brings me to my opinion. git is a tool in its own right. It can be used in the repository straight away. You not only don't need projen for it, but projen doesn't even attempt to claim to be the main entrypoint for it. Thus .gitignore is required to be clones for it to be behaving as intended by the git devs. Note that git still works without it. "Making sure certain files never end up in the repo" is the feature here that is impacted by .gitignore not being available. This is contrary to tools like tsc, prettier or eslint which are very much deeply integrated into the projen workflows. Sure you can run tsc directly. But the intended way is to go via projen build and that will take care of the requirements.

Unfortunately this doesn't lead us to a universal definition yet. The best I can come with so far is something along the lines of this: "Unless it results in something not working, or not behaving as intended without further safeguards" is the best definition I can come up with for now.