cdklabs / jsii-docgen

Generates reference documentation for jsii modules

jsii-docgen

Generates markdown reference documentation for jsii modules.

Usage

You can use this as a command line tool or as a library if you need more control.

$ jsii-docgen

Will produce a file called API.md with the api reference for this module.

As a library:

import { Documentation, Language } from 'jsii-docgen';

const docs = await Documentation.forProject('.');
const markdown = await docs.toMarkdown({ language: Language.TYPESCRIPT }).render(); // returns a markdown string

const json = await docs.toJson({ language: Language.TYPESCRIPT }).render(); // returns a JSON object

Curreently jsii-docgen supports generating documentation in the following languages:

• TypeScript (typescript)
• Python (python)
• Java (java)
• C# (csharp)
• Go (go)

CLI Options

Option	Required	Description
--output, -o	optional	Output filename (defaults to API.md if format is markdown, and API.json if format is JSON). jsii-docgen -o ./docs/API.md
--format, -f	optional	Output format. Can be markdown or json. jsii-docgen -f json
--language, -l	optional	Language to generate documentation for. Can be typescript, python, java, csharp or go. jsii-docgen -l typescript
--package, -p	optional	The name@version of an NPM package to document. jsii-docgen -p my-package
--readme, -r	optional	Generate docs for user specified README.md. jsii-docgen -r
--submodule, -s	optional	Generate docs for a specific submodule or "root". jsii-docgen -s my-submodule
--split-by-submodule	optional	Generate a separate file for each submodule. jsii-docgen --split-by-submodule

Contributions

All contributions are welcome, just raise an issue or submit a PR. Add a test, update readme. Do the right thing.

License

Apache 2.0

jsii-docgen's Issues

Annotate inherited methods

It may be useful to the reader of an API reference if it showed which methods were inherited from a superclass (and perhaps even which class it was inherited from?)

This might require augmenting the JSON schema for methods / callables with an additional field containing inheritance information.

Show code declaration of properties

Currently the Properties section of a class only displays the property name:

---

Properties

apiGroup^Required

• Type: string

The group portion of the API version (e.g. authorization.k8s.io).

---

In typescript and python this is ok since accessing this properties is done by name. This means we provide all the necessary programatic symbols in order to use the property.

However, in Java, properties need to be accessed via a getXXX method. For full accuracy, we need to convert properties into java methods, and display them in the Methods section of a class.

However, this will dirty up the Methods section with a bunch of getters that don't really play the same role as normal methods. Even in Java, the concept of a property exists, and one could also theoretically declare a public property, in which case the current implementation would hold. Since JSII converts public typescript properties to getter methods (as it should), we find ourselves in this predicament.

Proposed solution (at least as a first step) is to show the declaration of a property as a code snippet of the property:

---

Properties

apiGroup^Required

public String getApiGroup();

• Type: string

The group portion of the API version (e.g. authorization.k8s.io).

---

This will provide all the information needed to use the property, as well as maintain the separation from "regular" methods.

Split pages into groups / classes

Hey there,

We are currently using jsii-docgen to generate our API documentation.
The problem is that it's all generated on one page, making it very hard to navigate. Having a way to bring a file-structure to the documentation, having a grouping either by class / folder / file / manual grouping would be helpful, it would make the docs more usable for readers.

Abbreviate primitive/collection Java types in code snippets

Minor enhancement.

It may improve the readability of the Java docs if we just show just the class name (String) for primitive types when the type is referenced within code snippets, but leave the the full Java import (java.lang.String) in the rest of the docs (outside of code snippets). For example:

CfnPublicRepositoryProps.builder()
//  .repositoryCatalogData(java.lang.Object)
//  .repositoryName(java.lang.String)
//  .repositoryPolicyText(java.lang.Object)
//  .tags(java.util.List<CfnTag>)
    .build();

should be:

CfnPublicRepositoryProps.builder()
//  .repositoryCatalogData(Object)
//  .repositoryName(String)
//  .repositoryPolicyText(Object)
//  .tags(List<CfnTag>)
    .build();

Invalid values for submodule in JSON output

I ran [email protected] on @aws-solutions-constructs/[email protected] with Java selected as the target language, and found this fragment:

{
  "fqn": "software.amazon.awsconstructs.services.lambdasqs.LambdaToSqs.Initializer.parameter.deadLetterQueueProps",
  "displayName": "deadLetterQueueProps",
  "id": "@aws-solutions-constructs/aws-lambda-sqs.LambdaToSqs.Initializer.parameter.deadLetterQueueProps",
  "optional": true,
  "default": "- Default props are used",
  "type": {
    "formattingPattern": "%",
    "types": [
      {
        "id": "aws-cdk-lib.aws_sqs.QueueProps",
        "displayName": "QueueProps",
        "fqn": "software.amazon.awscdk.services.sqs.QueueProps",
        "packageName": "aws-cdk-lib",
        "packageVersion": "2.8.0",
        "submodule": "software.amazon.awscdk.services.sqs"
      }
    ]
  },
  "docs": {
    "summary": "Optional user provided properties for the dead letter queue."
  }
},

Based on how the schema was designed, submodule should be equal to aws_sqs (the language-independent module name), not software.amazon.awscdk.services.sqs.

Variadic arguments are not treated as such

For example, a method signature like so:

public add(...scopes: IDependable[])

Translates into:

public void add(IDependable scopes)

This is true for all languages.

See https://constructs.dev/packages/constructs/v/10.0.98/api/DependencyGroup?lang=java for the example.

Docgen sometimes produces no output

In certain installs, the docgen doesn't produce anything.

I did some digging, and found this is because (in certain installs) the glob-promise callback never returns.

This is because glob-promise has a peer dependency of '*', which meant it is possible for it to use glob 9.x and later (which doesn't support callbacks) - resulting in no output from jsii-docgen.

I'll open a PR to fix this.

README does not get transliterated when specifying a submodule

Running the following code does not transliterate README.md:

await docs.toMarkdown({
  language: docgen.Language.fromString(language),
  submodule: 'some-submodule',
  readme: true,
});

However the following code works just fine:

await docs.toMarkdown({
  language: docgen.Language.fromString(language),
  allSubmodules: true,
  readme: true,
});

Add an option to generate a table of contents for large API references

This was available in jsii-docgen v1 but I think the enhancement was removed in v2. On https://constructs.dev/ we provide a table of contents automatically, but adding a table of contents within the generated markdown would make it easier to see all of the classes, structs, enums etc. that are available at a glance when rendering the docs on GitHub, GitLab, etc.

Example: https://github.com/projen/projen/blob/v0.46.0/API.md

Cannot create docgen on a AwsCdkConstructLibrary from projen

Hi, I'm trying to create a AwsCdkConstructLibrary with projen. This fails at a stage and I think the bug is down to a bug in jsii-docgen.

When running yarn jsii-docgen on the latest version, I get the following error:

Error: Invalid assembly:
0: instance.schema is not one of enum values: jsii/0.10.0
1: instance requires property "author"
2: instance requires property "description"
3: instance requires property "fingerprint"
4: instance requires property "homepage"
5: instance requires property "jsiiVersion"
6: instance requires property "license"
7: instance requires property "name"
8: instance requires property "repository"
9: instance requires property "version"

    at Object.validateAssembly (MY_FOLDER/node_modules/@jsii/spec/lib/validate-assembly.js:15:11)
    at Assembly.validate (MY_FOLDER/node_modules/jsii-reflect/lib/assembly.js:158:14)
    at TypeSystem._loadModule (MY_FOLDER/node_modules/jsii-reflect/lib/type-system.js:92:21)
    at async TypeSystem._loadModule (MY_FOLDER/node_modules/jsii-reflect/lib/type-system.js:107:17)
    at async TypeSystem.loadModule (MY_FOLDER/node_modules/jsii-reflect/lib/type-system.js:65:21)
    at async Object.renderSinglePageModule (MY_FOLDER/node_modules/jsii-docgen/lib/index.js:41:17)
    at async main (MY_FOLDER/node_modules/jsii-docgen/lib/cli.js:16:9)
error Command failed with exit code 1.

I've looked at the input that the files receive, and validate-assemply.js is receiving

{
  schema: 'jsii/file-redirect',
  compression: 'gzip',
  filename: '.jsii.gz'
}

for the folder node_modules/aws-cdk-lib/package.json.

Projen, for example, unzips first the file before processing https://github.com/projen/projen/blob/ebd75ce77014cdb0c0daedf1b6de7b1f3157ecf9/src/inventory.ts#L118

docgen for python flattens out structs that aren't flattened in codegen

The python documentation generated for this code:

export interface SecretValue {
  readonly secret: ISecret;
  readonly key: string;
}

export interface EnvValueFromSecretOptions {
  readonly optional?: boolean;
}

public static fromSecretValue(secretValue: secret.SecretValue, options: EnvValueFromSecretOptions = {}): EnvValue {
  ...
  ...
}

is this:

cdk8s_plus_25.EnvValue.from_secret_value(
  key: str,
  secret: ISecret,
  optional: bool = None
)

Notice how it flattened the first struct argument, even though its not supposed to. The docs should instead be:

cdk8s_plus_25.EnvValue.from_secret_value(
  secret_value: typing.Union["SecretValue", typing.Dict[builtins.str, typing.Any]],
  optional: bool = None
)

originally discussed here: aws/jsii#3883

--readme option documentation and implementation do not align

when running the CLI without the -r flag, it defaults to true but this shouldn't be the case according to the documentation.

jsii-docgen/src/docgen/view/documentation.ts

Lines 251 to 253 in a159e5f

	if (options?.readme ?? true) {
	readme = new Readme(transpile, assembly, submodule).render();
	}

Are you accepting contribution for this?

Methods and properties from superclasses are not generated in docs

In the AWS CDK, the class aws_s3.Bucket extends from aws_s3.BucketBase, and correspondingly inherits several methods and properties. These methods and properties are currently not included in the generated docs, even though they should be.

CLI command only generates docs in root submodule by default

JSII projects may have multiple submodules (for examples libraries, see aws-cdk-lib and projen). Currently the CLI only generates docs for the root submodule by default. This appears to be a regression from jsii docgen v0.1.x.

I think the default behavior for both the programmatic API and the CLI should be to render docs for all submodules if no other option has been specified. There should also be a way in the CLI to specify specific submodules to render (to match the submodule option available in the programmatic API).

Classes with same name don't have differentiating h3 anchor tags.

In a project with the following setup:

.
-- namespace1
-- SomeClass
-- namespace2
-- SomeClass

Will result in an API.md where both h3 anchors have the same name:

### SomeClass <a name="SomeClass" id="namespace1.SomeClass"></a>
### SomeClass <a name="SomeClass" id="namespace2.SomeClass"></a>

When rendering these docs (using something like docusaurus) the names are not distinct enough and causes confusion as namespaces are not printed anywhere.

Multi-lines formatting issue in markdown

Jsii-docgen generates API documentation with only 2 lines: the first sentence and the rest of the code comment.

It generates incorrect output in various cases including:

• Only the first comment line is interpreted with a CR. The other CRs are not interpreted
• Code block is one single line, no indentation
• bullet lists are not formatted properly but all items are put into the same line with "*"

Example here :

• The code comment https://github.com/aws-samples/aws-analytics-reference-architecture/blob/d9f06ff7127754368ee1f6a9ffa472750c04c7d4/core/src/data-lake-storage.ts#L50
• The generated API.md https://github.com/aws-samples/aws-analytics-reference-architecture/blob/main/core/API.md#datalakestorage-

Go Language Support

Add a GoTranspile class to support generating JSII module documentation for Go.

When using split-by-submodule, documentation for the root of the project is not generated

Currently using jsii-docgen with projen and would like to have docs split up by submodule, using the --split-by-submodule option. However, when using this, no documentation is produced for the constructs in the root of the project.

The full command:

$  npx jsii-docgen .jsii -o docusaurus/docs/api/api --split-by-submodule

With this, I get an 'api.md' file in the docusaurus/docs/api directory that has the following contents:

# Submodules <a name="Submodules" id="submodules"></a>

The following submodules are available:
- [awscdk](./awscdk.md)
- [build](./build.md)
- [cdk](./cdk.md)
- [cdk8s](./cdk8s.md)
- [cdktf](./cdktf.md)
- [circleci](./circleci.md)
- [github](./github.md)
- [gitlab](./gitlab.md)
- [java](./java.md)
- [javascript](./javascript.md)
- [python](./python.md)
- [release](./release.md)
- [typescript](./typescript.md)
- [vscode](./vscode.md)
- [web](./web.md)

But none of those files contain any of the documentation for files in the 'src' directory, like gitpod.ts, ignore-file.ts, and others.

How can I have smaller, more fine-grained documentation files, without losing documentation for constructs in the root of the project?

Create Open API schema based on json output

Similar to how decdk has a schema that it can generate for users (https://github.com/cdklabs/decdk/#getting-started)

`[object Object]` in generated documentation

[object Object]... Looks like you're good to file an issue against jsii-docgen here...

Originally posted by @RomainMuller in cdklabs/construct-hub#174 (comment)

API.md empty

I have a projen construct library, and at some point I upgraded packages and my API.md turned into just the title
# API Reference <a name="API Reference"></a>
and 5 blank lines.

Even after updating to latest I am still unable generate an API.md

This is the change that seemed to cause the issue:
yarnlockdiff.txt

   "devDependencies": {
-    "@aws-cdk/assert": "1.116.0",
+    "@aws-cdk/assert": "1.117.0",
     "@types/jest": "^26.0.24",
     "@types/node": "^10.17.0",
-    "@typescript-eslint/eslint-plugin": "^4.28.4",
-    "@typescript-eslint/parser": "^4.28.4",
-    "esbuild": "^0.12.15",
-    "eslint": "^7.31.0",
+    "@typescript-eslint/eslint-plugin": "^4.29.0",
+    "@typescript-eslint/parser": "^4.29.0",
+    "esbuild": "^0.12.18",
+    "eslint": "^7.32.0",
     "eslint-config-prettier": "^8.3.0",
     "eslint-import-resolver-node": "^0.3.4",
     "eslint-import-resolver-typescript": "^2.4.0",
@@ -45,104 +42,103 @@
     "eslint-plugin-prettier": "^3.4.0",
     "jest": "^27.0.6",
     "jest-junit": "^12",
-    "jsii": "^1.31.0",
-    "jsii-diff": "^1.31.0",
-    "jsii-docgen": "^1.8.110",
-    "jsii-pacmak": "^1.31.0",
-    "jsii-release": "^0.2.67",
+    "jsii": "^1.32.0",
+    "jsii-diff": "^1.32.0",
+    "jsii-docgen": "^3.2.19",
+    "jsii-pacmak": "^1.32.0",
+    "jsii-release": "^0.2.82",
     "json-schema": "^0.3.0",
     "npm-check-updates": "^11",
     "prettier": "^2.3.2",
-    "projen": "^0.17.95",
-    "standard-version": "^9",
+    "projen": "^0.27.18",
     "ts-jest": "^27.0.4",
     "typescript": "^4.3.5"
   }```

Support for JSDoc Block Tags

Looking at the CDK official documentation, I noticed there are a number of UI elements produced by JSDoc block tags within its documentation that do not appear in the output of the markdown generated by this library. Running some experiments, I have gathered the following:

tag	supported?	notes
@deprecated	partial	⚠️ symbol is rendered, but text is not
@default	unsupported
@see	unsupported

Am I correct that these are not handled by jsii-docgen? Is there any way to get these values in the outputted markdown?

An aside: I am realizing I may have been incorrect in assuming that this library is what was used to generate the CDK online documentation. Is there another library used or is this library heavily modified for that purpose?

Java Support

Generate API reference in Java

Error: unable to determine assembly since module does not have any types

Hi,
I have a problem with generating API documentation with jsii-docgen for the python library.
Given module layout:

src/
  core/several_files.ts
  addons/
    aws/
      index.ts
    index.ts
  index.ts

The following problem happens as soon as I add a named export on the second level of the library.

# This works!
src/index.ts
export * from "./core"
export * as addons from "./addons"
# As soon as adding follwoing to the src/addons/index.ts
export * as aws from "./aws"

Error message:

 npx projen && npx projen compile && npx projen package-all && rm -f docs/api/api.*.md && npx projen docs:api
👾 default | ts-node --project tsconfig.dev.json .projenrc.ts
👾 Installing dependencies...
👾 install | yarn install --check-files
yarn install v1.22.19
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
success Already up-to-date.
✨  Done in 2.77s.
👾 Installing dependencies...
👾 install | yarn install --check-files
yarn install v1.22.19
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
success Already up-to-date.
✨  Done in 3.45s.
👾 compile | jsii --silence-warnings=reserved-word
👾 package-all » package:js | jsii-pacmak -v --target js
[jsii-pacmak] [INFO] Found 1 modules to package
[jsii-pacmak] [INFO] Packaging NPM bundles
[jsii-pacmak] [INFO] Loading jsii assemblies and translations
[jsii-pacmak] [INFO] Packaging 'js' for @gcix/gcix
[jsii-pacmak] [INFO] js finished
[jsii-pacmak] [INFO] Packaged. npm pack (37.2s) | js (0.1s) | load jsii (0.0s) | cleanup (0.0s)
👾 package-all » package:python | jsii-pacmak -v --target python
[jsii-pacmak] [INFO] Found 1 modules to package
[jsii-pacmak] [INFO] Packaging NPM bundles
[jsii-pacmak] [INFO] Loading jsii assemblies and translations
[jsii-pacmak] [INFO] Packaging 'python' for @gcix/gcix
[jsii-pacmak] [INFO] python finished
[jsii-pacmak] [INFO] Packaged. npm pack (34.2s) | python (20.5s) | load jsii (0.0s) | cleanup (0.0s)
👾 docs:api » compile | jsii --silence-warnings=reserved-word
👾 docs:api | npx jsii-docgen -l typescript -l python --readme false --output ./docs/api/api
Error: unable to determine assembly since module does not have any types: @gcix/gcix.addons
    at PythonTranspile.getParentModule (/Users/danielvonessen/git/gitlab/dvonessen/gcix/node_modules/jsii-docgen/lib/docgen/transpile/transpile.js:393:19)
    at PythonTranspile.moduleLike (/Users/danielvonessen/git/gitlab/dvonessen/gcix/node_modules/jsii-docgen/lib/docgen/transpile/python.js:240:35)
    at PythonTranspile.type (/Users/danielvonessen/git/gitlab/dvonessen/gcix/node_modules/jsii-docgen/lib/docgen/transpile/python.js:219:33)
    at /Users/danielvonessen/git/gitlab/dvonessen/gcix/node_modules/jsii-docgen/lib/docgen/view/interface.js:16:81
    at Array.map (<anonymous>)
    at new Interface (/Users/danielvonessen/git/gitlab/dvonessen/gcix/node_modules/jsii-docgen/lib/docgen/view/interface.js:16:57)
    at /Users/danielvonessen/git/gitlab/dvonessen/gcix/node_modules/jsii-docgen/lib/docgen/view/interfaces.js:9:25
    at Array.map (<anonymous>)
    at new Interfaces (/Users/danielvonessen/git/gitlab/dvonessen/gcix/node_modules/jsii-docgen/lib/docgen/view/interfaces.js:9:14)
    at new ApiReference (/Users/danielvonessen/git/gitlab/dvonessen/gcix/node_modules/jsii-docgen/lib/docgen/view/api-reference.js:30:27)
👾 Task "docs:api" failed when executing "npx jsii-docgen -l typescript -l python --readme false --output ./docs/api/api" (cwd: /Users/USERNAME/git/gitlab/USERNAME/gcix)

The repo can be found at https://gitlab.com/gcix/gcix.

Errors while generating docs for aws-cdk v1.126.0

When trying to generate docs for the experimental submodule of @aws-cdk/[email protected] we get the following errors:

Error: XXX is not a supported target for module: @aws-cdk/aws-cloudfront.experimental

Repro:

const docgen = require('jsii-docgen');


async function main() {
  const docs = await docgen.Documentation.forPackage('@aws-cdk/[email protected]', {
    language: docgen.Language.PYTHON,
  });

  console.log(docs.render({
    submodule: 'experimental'
  }).render());
}

main().catch(e => {
  console.log(e);
  process.exit(1);
});

Reporting a vulnerability

Hello!

I hope you are doing well!

We are a security research team. Our tool automatically detected a vulnerability in this repository. We want to disclose it responsibly. GitHub has a feature called Private vulnerability reporting, which enables security research to privately disclose a vulnerability. Unfortunately, it is not enabled for this repository.

Can you enable it, so that we can report it?

Thanks in advance!

PS: you can read about how to enable private vulnerability reporting here: https://docs.github.com/en/code-security/security-advisories/repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository

Error: Invalid assembly * must be string

The build 7.0.56 (use jsii's loadAssemblyFromFile function #723) causes the above error when using lerna v5.3.0

👾 build » post-compile | lerna run post-compile --stream
lerna notice cli v5.3.0
lerna info versioning independent
lerna info Executing command in 1 package: "npm run post-compile"
constructone: > [email protected] post-compile
constructone: > npx projen post-compile
constructone: 👾 post-compile » docgen | jsii-docgen -o API.md
constructone: Error: Invalid assembly:
constructone:  * must be string
constructone:     at validateAssembly (/home/ec2-user/development/patternlibrary/lp2/constructone/node_modules/@jsii/spec/lib/validate-assembly.js:12:15)
constructone:     at loadAssemblyFromBuffer (/home/ec2-user/development/patternlibrary/lp2/constructone/node_modules/@jsii/spec/lib/assembly-utils.js:74:64)
constructone:     at loadAssemblyFromFile (/home/ec2-user/development/patternlibrary/lp2/constructone/node_modules/@jsii/spec/lib/assembly-utils.js:100:12)
constructone:     at /home/ec2-user/development/patternlibrary/lp2/constructone/node_modules/jsii-docgen/lib/docgen/view/documentation.js:222:62
constructone:     at async withTempDir (/home/ec2-user/development/patternlibrary/lp2/constructone/node_modules/jsii-docgen/lib/docgen/view/documentation.js:271:16)
constructone:     at async Documentation.createAssembly (/home/ec2-user/development/patternlibrary/lp2/constructone/node_modules/jsii-docgen/lib/docgen/view/documentation.js:212:25)
constructone:     at async Documentation.toJson (/home/ec2-user/development/patternlibrary/lp2/constructone/node_modules/jsii-docgen/lib/docgen/view/documentation.js:125:28)
constructone:     at async Documentation.toMarkdown (/home/ec2-user/development/patternlibrary/lp2/constructone/node_modules/jsii-docgen/lib/docgen/view/documentation.js:170:23)
constructone:     at async main (/home/ec2-user/development/patternlibrary/lp2/constructone/node_modules/jsii-docgen/lib/cli.js:51:21)
constructone: 👾 Task "post-compile » docgen" failed when executing "jsii-docgen -o API.md" (cwd: /home/ec2-user/development/patternlibrary/lp2/constructone)
lerna ERR! npm run post-compile exited 1 in 'constructone'
👾 Task "build » post-compile" failed when executing "lerna run post-compile --stream" (cwd: /home/ec2-user/development/patternlibrary/lp2)

(tech debt) Remove HEADER_SPAN hack

The markdown renderer has a small hack when rendering headers that allows the "HEADING_SPAN" environment variable to be passed to replace all anchors with <span> tags with data-attribute properties. I think this hack exists so that anchors can be rendered properly on Construct Hub using the prism-react-renderer component.

Once the Construct Hub has switched to rendering API references directly from API.json files, we can remove this hack.

Cannot find module './jsii-docgen.js'

Hi @eladb

Looks like the jsii-docgen.js was not bundled in the npm package.

npx jsii-docgen                                                                                                                                                       master 
npx: 38 安裝成功，花費 4.275 秒
Cannot find module './jsii-docgen.js'

Render code snippets in library consumer format

Applies to all languages.

Currently, code examples for classes, constructs, structs, and static functions demonstrate how the APIs are used if a user is using the library (e.e. by showing object instantiation). However, instance methods currently show just their signatures.

To be more consistent, code snippets for methods should also show how they get used/invoked. In code, this might mean assuming an instance exists and then calling a method on it, for example:

// <impports>

CfnPublicRepository cfnPublicRepository;
cfnPublicRepository.inspect(TreeInspector inspector);

for Java, and something similar for TypeScript and Python.

Python generation fails on some packages

For example:

test('@aws-cdk/aws-route53-patterns', async () => {
  const docs = await Documentation.forPackage('@aws-cdk/[email protected]', {
    language: Language.PYTHON,
  });
  const markdown = docs.render({ linkFormatter: (t: TranspiledType) => `#custom-${t.fqn}` });
  expect(markdown.render()).toMatchSnapshot();
});

Debug Failure. False expression: Containing file should be a module.

      at checkImportMetaProperty (node_modules/typescript/lib/typescript.js:59527:22)
      at checkMetaProperty (node_modules/typescript/lib/typescript.js:59502:24)
      at checkExpressionWorker (node_modules/typescript/lib/typescript.js:61722:28)
      at checkExpression (node_modules/typescript/lib/typescript.js:61614:38)
      at checkNonNullExpression (node_modules/typescript/lib/typescript.js:57015:37)
      at checkPropertyAccessExpression (node_modules/typescript/lib/typescript.js:57060:85)
      at checkExpressionWorker (node_modules/typescript/lib/typescript.js:61693:28)
      at checkExpression (node_modules/typescript/lib/typescript.js:61614:38)
      at maybeCheckExpression (node_modules/typescript/lib/typescript.js:60827:34)
      at checkBinaryExpression (node_modules/typescript/lib/typescript.js:60795:25)
      at checkExpressionWorker (node_modules/typescript/lib/typescript.js:61734:28)
      at checkExpression (node_modules/typescript/lib/typescript.js:61614:38)
      at checkExpressionStatement (node_modules/typescript/lib/typescript.js:64083:13)
      at checkSourceElementWorker (node_modules/typescript/lib/typescript.js:66611:28)
      at checkSourceElement (node_modules/typescript/lib/typescript.js:66489:17)
      at Object.forEach (node_modules/typescript/lib/typescript.js:317:30)
      at checkSourceFileWorker (node_modules/typescript/lib/typescript.js:66826:20)
      at checkSourceFile (node_modules/typescript/lib/typescript.js:66794:13)
      at getDiagnosticsWorker (node_modules/typescript/lib/typescript.js:66882:17)
      at getDiagnostics (node_modules/typescript/lib/typescript.js:66868:24)
      at Object.getEmitResolver (node_modules/typescript/lib/typescript.js:35702:13)
      at node_modules/typescript/lib/typescript.js:101210:69
      at runWithCancellationToken (node_modules/typescript/lib/typescript.js:100921:24)
      at getDeclarationDiagnosticsForFileNoCache (node_modules/typescript/lib/typescript.js:101209:20)
      at getAndCacheDiagnostics (node_modules/typescript/lib/typescript.js:101222:26)
      at getDeclarationDiagnosticsWorker (node_modules/typescript/lib/typescript.js:101206:20)
      at getDeclarationDiagnosticsForFile (node_modules/typescript/lib/typescript.js:101235:56)
      at getDiagnosticsHelper (node_modules/typescript/lib/typescript.js:100860:24)
      at Object.getDeclarationDiagnostics (node_modules/typescript/lib/typescript.js:100905:24)
      at new SnippetTranslator (node_modules/jsii-rosetta/lib/translate.ts:154:20)
      at Translator.translatorFor (node_modules/jsii-rosetta/lib/translate.ts:91:24)
      at Translator.translate (node_modules/jsii-rosetta/lib/translate.ts:59:29)
      at Rosetta.translateSnippet (node_modules/jsii-rosetta/lib/rosetta.ts:165:39)
      at markdown_1.transformMarkdown.replace_typescript_transform_1.ReplaceTypeScriptTransform (node_modules/jsii-rosetta/lib/rosetta.ts:192:33)
      at ReplaceTypeScriptTransform [as replacer] (node_modules/jsii-rosetta/lib/markdown/replace-typescript-transform.ts:39:14)
      at ReplaceTypeScriptTransform.code_block (node_modules/jsii-rosetta/lib/markdown/replace-code-renderer.ts:15:22)
      at visitCommonMarkTree (node_modules/jsii-rosetta/lib/markdown/markdown.ts:71:21)
      at visitCommonMarkTree (node_modules/jsii-rosetta/lib/markdown/markdown.ts:73:5)
      at Object.transformMarkdown (node_modules/jsii-rosetta/lib/markdown/markdown.ts:11:5)
      at Rosetta.translateSnippetsInMarkdown (node_modules/jsii-rosetta/lib/rosetta.ts:188:12)
      at Object.transliterateAssembly (node_modules/jsii-rosetta/lib/commands/transliterate.ts:71:42)

Return types are not displayed in generated docs

The generated documentation does not display return types for any callables, instance methods or static methods.

For example:

This does not indicate that this static method returns an instance of AssetCode as is indicated by the signature:

public static fromAsset(path: string, options?: AssetOptions): AssetCode

Referencing external modules?

I'm trying utilize jsii-docgen to generate documentation for some custom CDK constructs I'm publishing. These constructs extend the Construct class of aws-cdk.core. If I attempt to generate documentation for only a single package's .jsii file (e.g.
npx jsii-docgen --output docs .jsii), an error is thrown Error: Assembly "@aws-cdk/core" not found (this is because core.Construct is the base of my construct).

I can generate documentation for all of my packages (e.g. npx jsii-docgen --output docs **/.jsii), however this generates (and links to) documentation for all of the CDK packages referenced within my constructs. I have no intention of publishing the CDK documentation, I'd prefer to either not link to any non-internal module or to perhaps link to the external CDK docs.

Are there any facilities available to support this? Or is my only solution to generate documentation for all referenced CDK classes and then delete it (and its references) in a post-processing step?

jsii-docgen doesn't find child jsii packages in a monorepo with npm workspaces

Hi there,

in an npm monorepo with the following structure:

packages/
├─ child-jsii-package/
│  ├─ package.json
├─ parent-jsii-package/
│  ├─ package.json   # includes child-jsii-package
package.json

jsii-docgen works fine on child-jsii-package, but fails on the parent-jsii-package with:

Error: Assembly "child-jsii-package" not found
    at TypeSystem.findAssembly (/Users/home/office/maintainer/contractcase/node_modules/jsii-reflect/lib/type-system.js:147:19)
    at TypeSystem.findFqn (/Users/home/office/maintainer/contractcase/node_modules/jsii-reflect/lib/type-system.js:156:26)
    at get type [as type] (/Users/home/office/maintainer/contractcase/node_modules/jsii-reflect/lib/type-ref.js:52:28)
    at TypeScriptTranspile.typeReference (/Users/home/office/maintainer/contractcase/node_modules/jsii-docgen/lib/docgen/transpile/transpile.js:317:17)
    at TypeScriptTranspile.parameter (/Users/home/office/maintainer/contractcase/node_modules/jsii-docgen/lib/docgen/transpile/typescript.js:134:30)
    at /Users/home/office/maintainer/contractcase/node_modules/jsii-docgen/lib/docgen/transpile/typescript.js:156:73
    at Array.map (<anonymous>)
    at TypeScriptTranspile.callable (/Users/home/office/maintainer/contractcase/node_modules/jsii-docgen/lib/docgen/transpile/typescript.js:156:35)
    at new InstanceMethod (/Users/home/office/maintainer/contractcase/node_modules/jsii-docgen/lib/docgen/view/instance-method.js:10:37)
    at /Users/home/office/maintainer/contractcase/node_modules/jsii-docgen/lib/docgen/view/instance-methods.js:9:25

Other jsii commands work (with the exception of this issue, which affects non-jsii packages marked with bundleDependencies, so my guess is it is a different cause - but I'm mentioning it in case it isn't).

Let me know if you need any further information

Method descriptions are missing in generated markdown files

Currently, method summaries (i.e. the first sentence in a method docstring) are included along with the table of methods. But the full description should also be shown alongside the actual method.

(screenshot of markdown as currently displayed on constructs.dev)

Notice above that no description for addCorsRule is shown (only descriptions of parameters are shown).

Classes do not correctly list all interfaces they implement

The AWS CDK documentation shows the interfaces that are implemented by a given class:

This is mirrored in jsii-docgen, except the information doesn't always match. For example on Construct Hub (which uses jsii-docgen under the hood), the page for CfnAccessPoint says that it only implements IInspectable (at the time of creating this issue) -- but the AWS CDK API Reference shows that it implements IInspectable, IConstruct, and IDependable.

My hunch is that there is a bug in how we're obtaining the list of interfaces -- perhaps the code is omitting interfaces implemented by superclasses, or omitting interfaces that are extended by other interfaces, etc.

Structs that allow any properties do not have any information shown

To see an example of this, visit https://constructs.dev/packages/construct-hub/v/0.3.192/ and go to the API reference for DenyListMap.

The page should show some kind of information to indicate that any field is allowed, per the original source code:

/**
 * The contents of the deny list file in S3.
 */
export interface DenyListMap {
  /**
   * A map from "name@version" to deny list rule.
   */
  readonly [key: string]: DenyListRule;
};

It looks like the root cause is that this information does not get saved in any form in the generated .jsii file (most other structs have a "properties" field with a list of the supported properties):

"construct-hub.DenyListMap": {
  "assembly": "construct-hub",
  "datatype": true,
  "docs": {
    "stability": "experimental",
    "summary": "The contents of the deny list file in S3."
  },
  "fqn": "construct-hub.DenyListMap",
  "kind": "interface",
  "locationInModule": {
    "filename": "src/backend/deny-list/api.ts",
    "line": 29
  },
  "name": "DenyListMap",
  "symbolId": "src/backend/deny-list/api:DenyListMap"
},

I think this is blocked until aws/jsii#2928 is resolved (since we would need a way to represent this kind of type in other languages).

Facing Error: Assembly "@custom-cdk-constructs/base-pipeline" not found

Currently we are exploring a way to generate documentation for our custom CDK construct.

We have created custom CDK contruct using following command
npx cdk init lib --language=typescript

and we tried to execute
npx jsii-docgen

And we faced following issue any help over here will be great.
Error: Assembly "@custom-cdk-constructs/base-pipeline" not found at TypeSystem.findAssembly (C:\Users\masked\AppData\Local\npm-cache\_npx\6d618b10e1acde92\node_modules\jsii-reflect\lib\type-system.js:147:19) at C:\Users\masked\AppData\Local\npm-cache\_npx\6d618b10e1acde92\node_modules\jsii-docgen\lib\docgen\view\documentation.js:235:23 at async withTempDir (C:\Users\masked\AppData\Local\npm-cache\_npx\6d618b10e1acde92\node_modules\jsii-docgen\lib\docgen\view\documentation.js:271:16) at async Documentation.createAssembly (C:\Users\masked\AppData\Local\npm-cache\_npx\6d618b10e1acde92\node_modules\jsii-docgen\lib\docgen\view\documentation.js:212:25) at async Documentation.toJson (C:\Users\masked\AppData\Local\npm-cache\_npx\6d618b10e1acde92\node_modules\jsii-docgen\lib\docgen\view\documentation.js:125:28) at async Documentation.toMarkdown (C:\Users\masked\AppData\Local\npm-cache\_npx\6d618b10e1acde92\node_modules\jsii-docgen\lib\docgen\view\documentation.js:170:23) at async main (C:\Users\masked\AppData\Local\npm-cache\_npx\6d618b10e1acde92\node_modules\jsii-docgen\lib\cli.js:51:21)

Do we need to do anything additional on our newly created construct apart from executing npx jsii-docgen

Render code snippets for enums

Missing imports in code snippets

Many of the code snippets refer to multiple types, but only generate a single import statement. This applies to all languages.

Example from cdk8s docs:

import { ApiObject } from 'cdk8s'

new ApiObject(scope: Construct, id: string, props: ApiObjectProps)

should include imports to Construct and ApiObjectProps.

Submodule name translations are not hierarchical

The submodules in the jsii assembly can map to language-specific names that are not hierarchical (they only need be nested under the root namespace, if relevant for the language). The Java code gen & general transpilation output data model have a baked in expectation that submodules represent a hierarchy, whereas they are just opaque labels.

This makes aws-cdk-lib (maybe) and monocdk (definitely) impossible to process through jsii-docgen for Java.

In particular, the below assumption does not hold water:

jsii-docgen/src/docgen/transpile/java.ts

Lines 69 to 71 in 20daf79

	if (!fqn.startsWith(parentFqn)) {
	throw new Error(`Expected submodule ${fqn} to start with ${parentFqn} since it is its parent module.`);
	}

Bug: Issue with temporary folder and symlinks

This issue occurs with the following project structure:

- root
  - packages
    - jsii project a
      - node_modules/project b -> symlink to project b containing '../../jsii project b'
    - jsii project b

jsii-reflect will correctly resolve all types. But jsii-docgen throws:
"Error: Assembly "jsii project b" not found"

This is (probably) due to the fact that the below code is making a full copy of the project structure:

jsii-docgen/src/docgen/view/documentation.ts

Line 180 in f3f0509

await fs.copy(assembliesDir, workdir);

The above code will copy the symlink as file to the temporary folder. This symlink will be invalid if its a relative link.

Probably either needs to fix it by not making a copy or to use something like:

await fs.copy(assembliesDir, workdir, {
    dereference: true
});

The copy also is responsible for a very long execution time due to all dependency copies. Didn't go through all code in the docgen package but it it really needed in this scenario to make a copy? I would think that the generator will not modify any files. Only the generation of the md file. Not copying would also make it possible to work without nohoist for peer dependencies.

This is actually part of a yarn monorepo where I configured the projects with nohoist. I cannot imagine i'm the only one encountering this. Anyone knows a way around it?

docgen running slow

version "6.3.23"

running jsii-docgen on my CDK libary project takes about 1.04 minutes The project itself isn't very big, it has some external dependencies though.
I'm wondering if that is expected, and if not, what would be a good way to figure out why it takes so long.

I have created a flame graph, and it seems to spend lot of time in copy.js and glob.js which hints to FS operations. (I can share the underlying data, its just too big to upload here)

Include jsii-rosetta examples

The AWS CDK API reference includes example code snippets for most constructs, classes, structs, and enums. Example. These are generated from information within jsii "tablet" files (if you install aws-cdk-lib, within the package you should find a .jsii.tabl.json file).

However, these are not displayed when you generate an API reference with jsii-docgen (and hence, they're not shown on Construct Hub). Example. We should find a way to include these since many users find them helpful given that CDK is vended in multiple languages.

Update jsii-rosetta, jsii-spec, jsii-reflect dependency to newer version 5.x.x

When using the dependency to jsii-docgen version 8.0.28 this pulls in "old" versions of jsii-spec, jsii-reflect and jsii-rosetta which then results in a warning when executing

jsii-docgen@^8.0.28:
version "8.0.28"
resolved "https://registry.yarnpkg.com/jsii-docgen/-/jsii-docgen-8.0.28.tgz#f327aa68a91ad7bce2af173809d72bcf2bf90174"
integrity sha512-h5+N5GKFHcydC87XeXcQjbmqM67zXqVx/iiAVQiUTNEhBkTW+j3sGL3TJ5pBRzNEev2MEJaAA9vlF0XS7iRhnQ==
dependencies:
"@jsii/spec" "^1.82.0"
case "^1.6.3"
fs-extra "^10.1.0"
glob "^8.1.0"
glob-promise "^6.0.2"
jsii-reflect "^1.82.0"
jsii-rosetta "^1.82.0"
semver "^7.5.1"
yargs "^16.2.0"

Lacking build steps

Cloning, running yarn and then yarn build does not result in a clean build. It fails during unit testing with the following error:

 FAIL  test/docgen/view/documentation.test.ts (347.879 s)

...

  ● package installation does not run lifecycle hooks

    Command failed: yarn package
    [jsii-pacmak] [WARN] Failed building python
    [jsii-pacmak] [WARN] python failed
    Error: Command (python3 -m venv --system-site-packages /tmp/npm-packGuKdUa/.env-NmWkkb) failed with status 1:
    #STDERR> 
    #STDOUT> The virtual environment was not created successfully because ensurepip is not
    #STDOUT> available.  On Debian/Ubuntu systems, you need to install the python3-venv
    #STDOUT> package using the following command.
    #STDOUT> 
    #STDOUT>     apt install python3.10-venv
    #STDOUT> 
    #STDOUT> You may need to use sudo with that command.  After installing the python3-venv
    #STDOUT> package, recreate your virtual environment.
    #STDOUT> 
    #STDOUT> Failing command: ['/tmp/npm-packGuKdUa/.env-NmWkkb/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']
    #STDOUT> 
    #STDOUT>

      49 |
      50 |   // create the package
    > 51 |   child.execSync('yarn package', { cwd: workdir });
         |         ^
      52 |
      53 |   // this should succeed because the failure script should be ignored
      54 |   const docs = await Documentation.forPackage(path.join(workdir, 'dist', 'js', `${libraryName}@0.0.0.jsii.tgz`), { name: libraryName });

      at ChildProcess.<anonymous> (../../../../tmp/4L5erf/node_modules/jsii-pacmak/lib/util.js:119:27)
error Command failed with exit code 1.
      at Object.<anonymous> (test/docgen/view/documentation.test.ts:51:9)

If there is something missing from the bootstrapping process I don't see it in the README and there is no CONTRIBUTING.md

It's not clear on how I can get all tests to pass.

Java code examples for generics with OR types do not render properly

Reproduction steps:

// script.ts (placed in the root of jsii-docgen)
import * as fs from 'fs';
import { Documentation, Language } from '.';

async function main() {
  const docs = await Documentation.forPackage('[email protected]', {
    language: Language.JAVA,
  });
  fs.writeFileSync('java.md', docs.render({ submodule: 'aws_ecr' }).render());
}

void main();

npx ts-node script.ts produces a markdown file which includes the following code snippet:

import software.amazon.awscdk.services.ecr.CfnReplicationConfiguration.ReplicationConfigurationProperty;

ReplicationConfigurationProperty.builder()
    .rules(IResolvable)
    .rules(java.util.List<ReplicationRuleProperty)
    .rules(IResolvable>)
    .build();

The correct code snippet should be:

import software.amazon.awscdk.services.ecr.CfnReplicationConfiguration.ReplicationConfigurationProperty;

ReplicationConfigurationProperty.builder()
    .rules(IResolvable)
    .rules(java.util.List<ReplicationRuleProperty OR IResolvable>)
    .build();

or (to be even more precise):

import software.amazon.awscdk.services.ecr.CfnReplicationConfiguration.ReplicationConfigurationProperty;

ReplicationConfigurationProperty.builder()
    .rules(IResolvable)
    .rules(java.util.List<ReplicationRuleProperty>)
    .rules(java.util.List<IResolvable>)
    .build();

jsii-docgen does not generate markdown for the submodules

If a typescript project contains an exported module (submodule for jsii), no information is generated in the markdown when jsii-docgen is ran against it.

Language Specific Nomenclature

Something to consider for jsii-docgen is that the nomenclature of "constructs, classes, structs, protocols, enums, etc" may not be terms that are familiar to users of a specific language. For example, Go obviously doesn't have classes, and "protocols" are a concept that map to Go interfaces, but constructs and classes are both also generated as Go interfaces in JSII.

This makes the docs generated for Go potentially unfamiliar to users of the language. Where they are expecting to see a bunch of free floating functions and struct/interface definitions, they are instead required to understand the constructs programming model and how that is translated to Go by JSII. This leakiness prevents providing an idiomatic experience in target languages as far as documentation is concerned.

Decrease test time / build time

There are a lot of tests for jsii-docgen, which will take longer to run as we add more languages. Currently they are taking more than 4 or 5 minutes on my development machine. I wonder if we can find some ways to optimize this.

One idea that comes to mind is to skip invoking jsii-rosetta during tests, since it does not really validate the core function of jsii-docgen and might be taking up a significant portion of the test time.