GithubHelp home page GithubHelp logo

kometenstaub / metadata-extractor Goto Github PK

View Code? Open in Web Editor NEW
98.0 2.0 6.0 159 KB

Obsidian Plugin that provides metadata export for use with third-party apps.

License: MIT License

TypeScript 90.78% JavaScript 9.22%
obsidianmd obsidian-plugin

metadata-extractor's Introduction

Metadata Extractor Obsidian plugin

This plugin allows you to write Obsidian vault metadata, which is only accessible via plugin, onto the hard drive. This enables Third-party apps to access Obsidian metadata which they normally wouldn't be able to access. Exemplary use cases are launcher apps (e.g. Alfred, Ulauncher) or graph analysis software.

See this guide for more information on Controlling Obsidian via a Third-party App.

There are four JSON-exports

They can be executed on a schedule.

Tag export

One writes a JSON file to disk with each tag and its corresponding file paths.

Example:

[
	{
		"tag": "css-themes",
		"tagCount": 5,
		"relativePaths": ["Advanced topics/Contributing to Obsidian.md"]
	},
	{
		"tag": "insider-build",
		"tagCount": 3,
		"relativePaths": ["Advanced topics/Insider builds.md"]
	},
	{
		"tag": "anothertag",
		"tagCount": 2,
		"relativePaths": [
			"Plugins/Zettelkasten prefixer.md",
			"Advanced topics/Using obsidian URI.md"
		]
	}
]

TypeScript interface:

/**
 * JSON export: tagToFile[]
 */
interface tagToFile {
	tag: string;
	tagCount: number;
	relativePaths: string[] | string;
};

Markdown notes metadata export

The second one writes a JSON file to disk with metadata for each file name. This is how the JSON structure is as a TypeScript interface.

/**
 * JSON export: Metadata[]
 */
import {extendedFrontMatterCache} from "./interfaces";

interface Metadata {
	fileName: string;
	relativePath: string;
	tags?: string[];
	headings?: { heading: string; level: number }[];
	aliases?: string[];
	links?: links[];
	backlinks?: backlinks[];
	frontmatter?: extendedFrontMatterCache;
}

interface links {
	link: string;
	relativePath?: string;
	cleanLink?: string;
	displayText?: string;
}

interface backlinks {
	fileName: string;
	link: string;
	relativePath: string;
	cleanLink?: string;
	displayText?: string;
}

interface extendedFrontMatterCache {
	cssclass?: string;
	publish?: boolean;
	position: Pos; // Pos is from the Obsidian API
	[key: string]: any;
}

The exported array contains a JSON array with Metadata objects, one object for each Markdown file in your vault.

All objects have a fileName and a relativePath. fileName doesn't contain the .md extension, relativePath is the path from your vault root.

If a file has tags, the object has a tags property that contains an array of tags. Tags are all lower-cased and if a tag appears more than one time in a file, it will only appear one time in tags. If a file has any frontmatter it is included in frontmatter. The structure of the object depends on your frontmatter.

aliases, links and backlinks also only exist if there are any of the in a file.

links interface

The links contain both links to existing and non-existing files. If a file doesn't exist, the links won't have a relativePath.

link is the full link, exluding anything after the |, so if no alias is set, it also contains # or #^ if there are headings or block references. If that is the case, there is also the cleanLink property which provides just the filename for the link (omitting the .md extension).

displayText is what is displayed by Obsidian in preview mode. It can be the alias, but also the file name if there is a heading or block reference. If it is a heading link or block reference to the same file, it excludes the #, just like Obsidian does in preview mode.

cleanLink and displayText don't exist if it is a normal link.

backlinks interface

Backlinks always have a relativePath property because the file linking to the current file (object) needs to exist.

fileName and relativePath are the file which contains the backlink.

link, cleanLink and displayText behave as the links interface

Non-Markdown files metadata export

The third writes a JSON file containing both all folders and non-Markdown files. The structure is like this.

/**
 * JSON export
 */
interface exceptMd {
	folders: folder[];
	nonMdFiles?: file[];
}

interface folder {
	name: string;
	relativePath: string;
}

interface file {
	name: string;
	basename: string;
	relativePath: string;
}

file interface

The name is the file name including the extension, basename excludes it. relativePath is the path from the vault root.

Canvas metadata export

The fourth export writes a JSON file containing name, basename and relativePath of canvas files as object in an array.

[
  {
    "name": "my-canvas.canvas",
    "basename": "my-canvas",
    "relativePath": "Inbox/my-canvas.canvas"
  },
  {
    "name": "visualisation.canvas",
    "basename": "visualisation",
    "relativePath": "visualisation.canvas"
  }
]

Configuration

If you don't touch any settings, the files will be saved to the plugin folder. You can configure their names in the settings.

You can however also specify absolute paths for each file. They need to include the file name and extension in this case. The setting above won't have any effect then.

You can also set the frequency for writing the JSON files in minutes (default setting is 0, so it is not enabled) and whether the JSON files should be written on launch (default setting is false).

metadata-extractor's People

Contributors

chrisgrieser avatar claremacrae avatar dennisseidel avatar dependabot[bot] avatar kometenstaub avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

metadata-extractor's Issues

Possible to maintain case?

Hello,

For my tags I use camel case to help distinguish mutli-word tags. The generated JSON strips all this formatting. Any chance we could keep it in there?

Cheers!

Sort metadata.json by recency

most recent access or most recent modification assorting criterion, with the most recent on top. This is information a launcher app cannot get easily, and which would help the launcher app to display the results in a way that that recent files are on top, which is more convenient to the user.

Problem with exported Backlinks

Hi,

first of all thank you for this great plugin! I`am using this in my solution Perlite to build the graph.

Unfortunately I have a strange problem with some backlinks. It seams that this problem is related to similar file names but I cant confirm this finally.

The exporter create backlinks to other files, where no backlink exists, also these backlings switches sometimes from one md to another, without doing any changes.

Check this example from the metadata.json.

"fileName": "WriteUp",
    "relativePath": "WriteUps/TryHackMe/Skynet/WriteUp.md",
    "backlinks": [
      {
        "fileName": "_Windows Priv Esc",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/Windows Related/_Windows Priv Esc.md",
        "displayText": "../../WriteUps/TryHackMe/Alfred/WriteUp"
      },
      {
        "fileName": "WinRM SSL Authentication",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/Windows Related/WinRM SSL Authentication.md",
        "displayText": "../../WriteUps/HackTheBox/Timelapse/WriteUp"
      },
      {
        "fileName": "Tokens",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/Windows Related/Tokens.md",
        "displayText": "../../WriteUps/TryHackMe/Alfred/WriteUp"
      },
      {
        "fileName": "SCF SMB URL Attack",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/Windows Related/SCF SMB URL Attack.md",
        "displayText": "../../WriteUps/HackTheBox/Driver/WriteUp"
      },
      {
        "fileName": "LAPS",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/Windows Related/LAPS.md",
        "displayText": "../../WriteUps/HackTheBox/Timelapse/WriteUp"
      },
      {
        "fileName": "SSTI",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/OWASP and Web Attacks/SSTI.md",
        "displayText": "../../WriteUps/HackTheBox/Spider/WriteUp"
      },
      {
        "fileName": "Reverse Tabnabbing",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/OWASP and Web Attacks/Reverse Tabnabbing.md",
        "displayText": "../../WriteUps/HackTheBox/Developer/WriteUp"
      },
      {
        "fileName": "Net Desirialization",
        "link": "Writeup",
        "relativePath": "OFFSEC Notes/OWASP and Web Attacks/Net Desirialization.md",
        "displayText": "../../WriteUps/HackTheBox/Sharp/Writeup"
      },
      {
        "fileName": "CORS",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/OWASP and Web Attacks/CORS.md",
        "displayText": "../../WriteUps/HackTheBox/Developer/WriteUp"
      },
      {
        "fileName": "IPMI",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/Network Services/IPMI.md",
        "displayText": "../../WriteUps/HackTheBox/Shibboleth/WriteUp"
      },
      {
        "fileName": "FTP",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/Network Services/FTP.md",
        "displayText": "../../WriteUps/HackTheBox/Pikaboo/WriteUp"
      },
      {
        "fileName": "Mongo and NoSQL",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/Databases/Mongo and NoSQL.md",
        "displayText": "../../WriteUps/HackTheBox/Secret/WriteUp"
      },
      {
        "fileName": "LDAP",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/Databases/LDAP.md",
        "displayText": "../../WriteUps/HackTheBox/Pikaboo/WriteUp"
      },
      {
        "fileName": "Perl",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/BASIC Fundamentals/Perl.md",
        "displayText": "../../WriteUps/HackTheBox/Pikaboo/WriteUp"
      },
      {
        "fileName": "OTP",
        "link": "WriteUp",
        "relativePath": "OFFSEC Notes/BASIC Fundamentals/OTP.md",
        "displayText": "../../WriteUps/HackTheBox/Static/WriteUp"
      }
    ]
  },

But actually there are no Backlinks to this file:
image

Thanks for looking into this!

BR,
sec77

Feature Request: add Kanban-yaml to metadata

if you could add the presence of this yaml key to the metadata, Alfred could detect whether a note is a kanban board or note. This would enable various features, like for example adding cards to Kanbans via Alfred.

---
kanban-plugin: basic
---

[BUG]: `metadata.json` cannot be written anymore

Based on the last creation date of a metadata.json, it stopped working on March 20th, so I assume the problem is not related to the latest release, but rather to some weird condition that changed in my files.

The other files, tags.json and allExceptMd.json, still work fine.

This is the log I get with "Show Console Logs" enabled:

app.js:1 Command failed to execute:  metadata-extractor:write-metadata-json
app.js:1 TypeError: Cannot read properties of undefined (reading 'frontmatter')
    at y.writeCacheToJSON (eval at <anonymous> (app.js:1:1444405), <anonymous>:33:2765)
    at Object.callback (eval at <anonymous> (app.js:1:1444405), <anonymous>:33:9951)
    at rz (app.js:1:1565835)
    at e.executeCommandById (app.js:1:1567080)
    at eval (eval at <anonymous> (app.js:1:1444405), <anonymous>:74:11668)
    at e.o [as executeCommandById] (eval at <anonymous> (app.js:1:1444405), <anonymous>:23:77225)
    at T.eval (eval at <anonymous> (app.js:1:1444405), <anonymous>:15:16522)
    at Generator.next (<anonymous>)
    at eval (eval at <anonymous> (app.js:1:1444405), <anonymous>:15:363)
    at new Promise (<anonymous>)
e.executeCommandById @ app.js:1
eval @ VM179:74
o @ VM179:23
eval @ VM165:15
eval @ VM165:15
e @ VM165:15
onChooseSuggestion @ VM165:15
eval @ VM165:15
eval @ VM165:15
e @ VM165:15
onChooseSuggestion @ VM165:15
t.selectSuggestion @ app.js:1
e.useSelectedItem @ app.js:1
(anonymous) @ app.js:1
e.handleKey @ app.js:1
e.onKeyEvent @ app.js:1

Enable trigger events

Is it possible to trigger this behaviour by invocation? For example on a pre-git commit or other manual trigger? Really cool plugin!

Reconsider implementation

if (!fullLink.includes('#')) {
currentLinkObject.link = fullLink;
// account for alias
if (aliasText !== fullLink) {
currentLinkObject.displayText = aliasText;
}
// account for uncreated files
if (relPath) {
currentLinkObject.relativePath = relPath.path;
}
}
// heading/block ref and maybe an alias, but not to the same file
else if (fullLink.includes('#') && fullLink.charAt(0) !== '#') {
const alias = aliasText;
const cleanLink = getLinkpath(fullLink);
currentLinkObject.link = fullLink;
currentLinkObject.cleanLink = cleanLink;
// it has an alias
if (!aliasText.includes('#') || !aliasText.includes('>')) {
currentLinkObject.displayText = alias;
}
// account for uncreated files
if (relPath) {
currentLinkObject.relativePath = relPath.path;
}
}
// heading/block ref to same file and maybe alias
else if (fullLink.charAt(0) === '#') {
currentLinkObject.link = fullLink;
currentLinkObject.relativePath = relativeFilePath;
currentLinkObject.cleanLink = displayName;
// account for alias
if (fullLink !== aliasText) {
currentLinkObject.displayText = aliasText;

https://github.com/obsidianmd/obsidian-api/blob/bceb489fc25ceba5973119d6e57759d64850f90d/obsidian.d.ts#L2618-L2627

https://github.com/obsidianmd/obsidian-api/blob/bceb489fc25ceba5973119d6e57759d64850f90d/obsidian.d.ts#L2359-L2367

Frontmatter export

What?

This is feature request to add an export of each file's YAML frontmatter to the note export. It looks like you've already got a reference to the frontmatter object, would just mean adding the whole thing as a key instead of/in addition to just the aliases.

Why?

My goal is to export and then further process notes to generate a static site blog from notes that meet the right criteria to be published. Basically just my own publish implementation that I maintain and deploy myself. Parsing the frontmatter using obisidan means hopefully less room for error in trying to parse it later when reading the file. I'm sure there are other applications.

Assuming that sounds good I might make a PR for this in the coming week when I have time, if you don't get to it first.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.