akosbalasko / yarle Goto Github PK

Option 1: leave resources folder as it is, put all the extracted content there, but we should replace the links to them in the MD files to what is configured.
Option 2: Upload them to the configured folder. In this case, what protocol should be supported, and the authentication should be solved as well.

If your note has an underscore like this:

And another note links to it like this:

Then yarle will create markdown files with escaped underscores:

# test - title_with_underscore

.

(Quelle: )

*   Zurück: 
    *   s
*   Weiter:
    *    s
*   Querverweise:
    *   s

    Created at: 2020-11-02T19:20:07+02:00
    Updated at: 2020-11-02T19:20:29+02:00
    Where: 29.18829748628224,65.96317843142039

# test - title_with_underscore2

[[test - title\_with\_underscore]]

(Quelle: )

*   Zurück:
    *   s
*   Weiter:
    *   s
*   Querverweise:
    *   s

    Created at: 2020-11-02T19:20:32+02:00
    Updated at: 2020-11-02T19:20:58+02:00
    Where: 29.18829748628224,65.963134765625

Other symbols might also get escaped. I haven't checked.

This is somewhat related to #73

It's going to be a MacOnly feature

As the title says: It's quite possible to have note titles with spaces in them and/or titles that are longer than 50 chars. This breaks internal links, since the title of the note needs to be the filename in order for internal links to work.

Related: #73, #90

The new --template option should be listed with the other options in the README, and the description of the option should contain a link ot the separate Templates.md document with detailed explanation which already exists.

Also, it should be possible to get the default template, so one can build a custom template based on that. The documentation shows a template that is "similar to YARLE's default template", but I would like to start with the exact same default template so that I don't miss anything.

Sorry to spam you with issues today but excited to try and get the library exported and having some issues

I've got lots of notes that have scientific article text pasted in. E.g. the below image

When run through your script this gets converted as follows:

Different roles in learning?

An obvious implication of the co-existence of multiple forms of LTP is that their roles in learning and memory processing might be dissociable. Tantalizing clues that this might, indeed, be the case are provided by a brief review of recent hippocampus-dependent behavior studies. With respect to LTP3, impaired function of L-type VDCCs seems to selectively impair long-term memory beyond a few days, without an effect on acquisition or working memory ![[67]], ![[78]] and ![[79]]. Similarly, interference with CRE-mediated transcription, the cAMP and PKA pathway and the ERK–MAPK pathway primarily affects long-term memory ![[2]] and ![[80]]. Correlating to LTP2, inhibition of Group I mGlu receptors seems to selectively impair reference memory during the period (24–48 h) between acquisition trials ![[81]] and ![[82]]. In relation to LTP1-specific mechanisms, genetic deletion of Ry receptors produces an intriguing behavior phenotype in which acquisition and long-term memory of a spatial learning task are intact, but the ability to subsequently learn a new arrangement of the same task is impaired ![[83]] and ![[84]]. The application of the LTP1, LTP2 and LTP3 framework could greatly assist in the design and interpretation of future investigations into the role of LTP in different aspects of learning and memory.

Obsidian then does something really wierd and converts this to the following:

Different roles in learning?

An obvious implication of the co-existence of multiple forms of LTP is that their roles in learning and memory processing might be dissociable. Tantalizing clues that this might, indeed, be the case are provided by a brief review of recent hippocampus-dependent behavior studies. With respect to LTP3, impaired function of L-type VDCCs seems to selectively impair long-term memory beyond a few days, without an effect on acquisition or working memory ![[unknown_filename.67.png]], ![[WhiskyHighland_Basic_Valuation_Port_Ellen_1978.pdf]] and ![[201907241020 heritability of alzheimers disease 79%]]. Similarly, interference with CRE-mediated transcription, the cAMP and PKA pathway and the ERK–MAPK pathway primarily affects long-term memory ![[201107121226 golgi]] and ![[201108180400 pw]]. Correlating to LTP2, inhibition of Group I mGlu receptors seems to selectively impair reference memory during the period (24–48 h) between acquisition trials ![[201107130819 jim eberwine, march 2011, neuron pnas, dec 2010]] and ![[201208261127 paid]]. In relation to LTP1-specific mechanisms, genetic deletion of Ry receptors produces an intriguing behavior phenotype in which acquisition and long-term memory of a spatial learning task are intact, but the ability to subsequently learn a new arrangement of the same task is impaired ![[201202010839 jobs to apply for_]] and ![[201201040845 earliest ferry from ryde on offpeak]]. The application of the LTP1, LTP2 and LTP3 framework could greatly assist in the design and interpretation of future investigations into the role of LTP in different aspects of learning and memory.

I don't really understand why Obsidian decides to swap e.g. ![[67]] for ![[WhiskyHighland_Basic_Valuation_Port_Ellen_1978.pdf]] but a good start might be to just not convert numeric values (even if they are hyperlinks) into [[...]] links. It seems to have might a right tangled mess of the knowledge graph as many notes have these!

Thanks again

Hi,

I'm trying to move from Evernote to Obsidian. I came across this in the Obsidian forum. I exported all my notes from Evernote to an Enex file. Then I set this running with:

npm run start -- --enexSource="/Users/natske/OneDrive - Imperial College London/Evernote Export/My Notes2.enex" --outputDir=./out --include-metadata --zettelkasten --outputFormat=ObsidianMD

It ran fine through the first 2675 notes then just stopped and didn't progress any further.

So it's been looking like that for 30 mins. Is there a debug log I could look in?

Thanks for developing this!

It would be nice (particularly for Obsidian) if file names would keep letter casing. Currently, everything seems to be converted to lower case.

Is there a reason why simple and complex notes are created in different folders? I'd like to have them merged in one because the distinction is only a technical one, not a logical one. I guess I can simply merge the folders, but then I may have to deal with duplicate file names.

I'm running macOS Catalina 10.15.5, and tried downloading the zip file.
I unzipped it, and from the yarle-master folder tried running 'npm i'
It gave a number of errors (see attached file)

'node -v' shows: v12.18.0

Any suggestions?
yarle-install.txt

Currently Obsidian itself do not support images as links hence, Yarle cannot support this feature yet.

I'd like to add some more configurable options to specify which kind of metadata should be added to md.

Thanks for writing this, it largely worked great, most files converted perfectly.

I had just a handful fail with errors. I put at least one of each error I received below.

Separate from fixing the errors, it'd be great if the app tracked these and printed a summary of those that failed to convert at the end of the process.

Failed to convert note: # Vehicle Identification Number VIN
TypeError: Cannot read property 'value' of undefined
at Object.replacement (/Users/jon/Dropbox/Development/Mac/yarle/dist/utils/turndown-rules/images-rule.js:12:40)
at TurndownService.replacementForNode (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:878:10)
at /Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:837:40
at NodeList.reduce ()
at TurndownService.process (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:830:17)
at TurndownService.replacementForNode (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:873:25)
at /Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:837:40
at NodeList.reduce ()
at TurndownService.process (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:830:17)
at TurndownService.replacementForNode (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:873:25)
Note # Vehicle Identification Number VIN
converted successfully.

Failed to convert note: # Stuff to bring to hotel for pub crawl
TypeError: Cannot read property 'value' of undefined
at Object.replacement (/Users/jon/Dropbox/Development/Mac/yarle/dist/utils/turndown-rules/task-items-rule.js:9:38)
at TurndownService.replacementForNode (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:878:10)
at /Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:837:40
at NodeList.reduce ()
at TurndownService.process (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:830:17)
at TurndownService.replacementForNode (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:873:25)
at /Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:837:40
at NodeList.reduce ()
at TurndownService.process (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:830:17)
at TurndownService.replacementForNode (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:873:25)
Note # Stuff to bring to hotel for pub crawl
converted successfully.

Another one was:
Failed to convert note: # Quotation for X
TypeError: ...
It was the whole file I believe.

Then:
is not a string, or an element/document/fragment node.
at TurndownService.turndown (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/turndown/lib/turndown.cjs.js:735:13)
at Object.exports.convertHtml2Md (/Users/jon/Dropbox/Development/Mac/yarle/dist/convert-html-to-md.js:5:65)
at Object.exports.processNode (/Users/jon/Dropbox/Development/Mac/yarle/dist/process-node.js:24:47)
at EventEmitter.xmlStream.on (/Users/jon/Dropbox/Development/Mac/yarle/dist/yarle.js:25:28)
at EventEmitter.emit (events.js:197:13)
at SAXStream.saxStream.on.tagName (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/xml-flow/lib/xml-flow.js:142:15)
at SAXStream.emit (events.js:197:13)
at SAXParser.me._parser.(anonymous function) [as onclosetag] (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/sax/lib/sax.js:258:17)
at emit (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/sax/lib/sax.js:624:35)
at emitNode (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/sax/lib/sax.js:629:5)

Failed to convert note: # Whatever Pictures
TypeError: Cannot read property '$text' of undefined
at processResource (/Users/jon/Dropbox/Development/Mac/yarle/dist/process-resources.js:43:32)
at Object.exports.processResources (/Users/jon/Dropbox/Development/Mac/yarle/dist/process-resources.js:14:79)
at Object.exports.processNode (/Users/jon/Dropbox/Development/Mac/yarle/dist/process-node.js:22:43)
at EventEmitter.xmlStream.on (/Users/jon/Dropbox/Development/Mac/yarle/dist/yarle.js:25:28)
at EventEmitter.emit (events.js:197:13)
at SAXStream.saxStream.on.tagName (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/xml-flow/lib/xml-flow.js:142:15)
at SAXStream.emit (events.js:197:13)
at SAXParser.me._parser.(anonymous function) [as onclosetag] (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/sax/lib/sax.js:258:17)
at emit (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/sax/lib/sax.js:624:35)
at emitNode (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/sax/lib/sax.js:629:5)
Note # Whatever Pictures
converted successfully.

Failed to convert note: # The psychology of
TypeError: Cannot read property '$text' of undefined
at processResource (/Users/jon/Dropbox/Development/Mac/yarle/dist/process-resources.js:43:32)
at Object.exports.processResources (/Users/jon/Dropbox/Development/Mac/yarle/dist/process-resources.js:19:75)
at Object.exports.processNode (/Users/jon/Dropbox/Development/Mac/yarle/dist/process-node.js:22:43)
at EventEmitter.xmlStream.on (/Users/jon/Dropbox/Development/Mac/yarle/dist/yarle.js:25:28)
at EventEmitter.emit (events.js:197:13)
at SAXStream.saxStream.on.tagName (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/xml-flow/lib/xml-flow.js:142:15)
at SAXStream.emit (events.js:197:13)
at SAXParser.me._parser.(anonymous function) [as onclosetag] (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/sax/lib/sax.js:258:17)
at emit (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/sax/lib/sax.js:624:35)
at emitNode (/Users/jon/Dropbox/Development/Mac/yarle/node_modules/sax/lib/sax.js:629:5)
Note # The psychology of
converted successfully.

I have a pretty involved nested tags tree in Evernote (the idea was to make a transition from EN easier). Example:

04-theorie.00-Begriffe.01-Philosophie.01-Logik

However, some Markdown software has different standards for nested tags:

• Bear uses slashes for tags: 04-theorie/00-Begriffe/01-Philosophie/01-Logik
• Obsidian can use either dashes or underscores to differentiate tags and subtags:
  • Dashes: 04-theorie-00-Begriffe-01-Philosophie-01-Logik
  • Underscores: 04-theorie_00-Begriffe_01-Philosophie_01-Logik

In order to fit the needs of people, we might want to offer the option to provide a replacement function that gets applied to each tag, if it is set.

For installation, the README states:

Installation

• install Node.js. Please use Node 10.18.1 because of 3rd party dependencies rebuild problems.

• clone or download this repo

• npm i

However, the 'build' directory is never created, leading to subsequent "npm run start" commands to fail.

npm run build appears to fix the issue.

Testing with v2.9.2 in Evernote I have some bits of text that link to normal https websites, in the output md file the links are missing.

Running using parameters: --include-metadata --outputFormat=ObsidianMD --skip-update-time --skip-creation-time

From Evernote:

I've confirmed this link is valid and works.

The output markdown file opened in BBEdit:

It would be nice to have the feature of templates to the converter, for instance a 'literature note' template known in Zettelkasten, to generate the notes directly on that format.

it would be nice to convert #anchornames from HTML to Obsidian as follows:
note.md

<h1>
<a href="#anchor"> anchorText</a>
</h1>

should be converted to

# [[note#anchor | anchorText]]

Each textbundle consists of three components:

1. info.json: it contains the metadata in the following form:

{
"creatorURL" : "file:///Applications/UlyssesMac.app/",
"transient" : false,
"type" : "net.daringfireball.markdown",
"creatorIdentifier" : "com.ulyssesapp.mac",
"version" : 2
}

2. text.md, in the following form:

# Untitled Note

![][image-1]![][image-2]

[image-1]: assets/Picture.jpg
[image-2]: assets/Picture-1.jp

3. optionally an assest subfolder that contains the assets fitting to the relative path in the text.

I have two notes that link to each other:

Test1+Test2.enex.zip

If I run the following: npm run start -- --enexSource=./in/Test1+Test2.enex --outputDir=./out/ --includeMetadata

The following set of Markdown files is created:

# Test1
.

(Quelle: )

*   Zurück:
    *   s
*   Weiter:
    *   s
*   Querverweise:
    *   <a href="evernote:///view/12512170/s114/b8eab9e6-e174-4eb5-8afe-08599a8762f2/b8eab9e6-e174-4eb5-8afe-08599a8762f2/">Test2</a>

# Test2
Test1

Test1abcd

Test123

[[Test1]]

(Quelle: )

*   Zurück:
    *   <a href="evernote:///view/12512170/s114/e65577b9-41dc-41ec-80ac-c6131597d983/e65577b9-41dc-41ec-80ac-c6131597d983/">Test1</a>
*   Weiter:
    *   s
*   Querverweise:
    *   s

out.zip

As you can see only one of the three links actually got converted correctly. See also #14 for the initial feature request.

It would be awesome if yarle could be used as a command that could be installed by simply running npm i -g yarle-evernote-to-md and run directly by doing npx yarle-evernote-to-md.

I took a look: What seems to be needed is a command file that needs to be specified in the bin field of the package.json.

Using the 2.9.2 release.

It seems to keep failing at the same place, so I checked the next note in the enex doc and it did contain some large attachments. After removing that note from the enex file, it was able to make it past that note, but it failed on another later on - I assume for the same reason (haven't checked it yet).

<--- Last few GCs --->

[53900:0x110008000] 53655 ms: Mark-sweep 1106.9 (1123.8) -> 1053.4 (1122.3) MB, 88.0 / 0.0 ms (average mu = 0.921, current mu = 0.768) allocation failure scavenge might not succeed
[53900:0x110008000] 53783 ms: Mark-sweep 1056.9 (1122.3) -> 1056.0 (1102.3) MB, 122.7 / 0.0 ms (average mu = 0.799, current mu = 0.044) allocation failure scavenge might not succeed

<--- JS stacktrace --->
Cannot get stack trace in GC.
FATAL ERROR: MarkCompactCollector: semi-space copy, fallback in old gen Allocation failed - JavaScript heap out of memory
1: 0x1011d1c65 node::Abort() (.cold.1) [/usr/local/bin/node]
2: 0x10009f919 node::Abort() [/usr/local/bin/node]
3: 0x10009fa7f node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
4: 0x1001e3867 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
5: 0x1001e3807 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
6: 0x10036b995 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
7: 0x1003c0127 v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/usr/local/bin/node]
8: 0x10039d7fc void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/usr/local/bin/node]
9: 0x10039d118 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/usr/local/bin/node]
10: 0x10039cd86 v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/usr/local/bin/node]
11: 0x1003c4dae v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/usr/local/bin/node]
12: 0x10038dd52 v8::internal::ItemParallelJob::Task::RunInternal() [/usr/local/bin/node]
13: 0x10038e1bf v8::internal::ItemParallelJob::Run() [/usr/local/bin/node]
14: 0x1003a0099 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/usr/local/bin/node]
15: 0x10039fb90 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/usr/local/bin/node]
16: 0x100392447 v8::internal::MarkCompactCollector::Evacuate() [/usr/local/bin/node]
17: 0x10038fe56 v8::internal::MarkCompactCollector::CollectGarbage() [/usr/local/bin/node]
18: 0x10036bcdf v8::internal::Heap::MarkCompact() [/usr/local/bin/node]
19: 0x100369533 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
20: 0x100367a3e v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
21: 0x10037390a v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
22: 0x100373991 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
23: 0x10034135a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/bin/node]
24: 0x100693768 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
25: 0x1009dcb79 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
[1] 53899 abort npm run start -- --outputDir=./jon/out --include-metadata

It is probably a good idea to document that that (on Windows) a Python installation is a prerequisite.
Otherwise this error occurs:

C:\A0\yarle-master\node_modules\node-expat>if not defined npm_config_node_gyp (node "C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
gyp ERR! find Python
gyp ERR! find Python Python is not set from command line or npm configuration
gyp ERR! find Python Python is not set from environment variable PYTHON
gyp ERR! find Python checking if "python" can be used
gyp ERR! find Python - "python" is not in PATH or produced an error
gyp ERR! find Python checking if "python2" can be used
gyp ERR! find Python - "python2" is not in PATH or produced an error
gyp ERR! find Python checking if "python3" can be used
gyp ERR! find Python - "python3" is not in PATH or produced an error
gyp ERR! find Python checking if the py launcher can be used to find Python 2
gyp ERR! find Python - "py.exe" is not in PATH or produced an error
gyp ERR! find Python checking if Python is C:\Python27\python.exe
gyp ERR! find Python - "C:\Python27\python.exe" could not be run
gyp ERR! find Python checking if Python is C:\Python37\python.exe
gyp ERR! find Python - "C:\Python37\python.exe" could not be run
gyp ERR! find Python
gyp ERR! find Python **********************************************************
gyp ERR! find Python You need to install the latest version of Python.
gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
gyp ERR! find Python you can try one of the following options:
gyp ERR! find Python - Use the switch --python="C:\Path\To\python.exe"
gyp ERR! find Python   (accepted by both node-gyp and npm)
gyp ERR! find Python - Set the environment variable PYTHON
gyp ERR! find Python - Set the npm configuration variable python:
gyp ERR! find Python   npm config set python "C:\Path\To\python.exe"
gyp ERR! find Python For more information consult the documentation at:
gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
gyp ERR! find Python **********************************************************
gyp ERR! find Python
gyp ERR! configure error
gyp ERR! stack Error: Could not find any Python installation to use
gyp ERR! stack     at PythonFinder.fail (C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\lib\find-python.js:307:47)
gyp ERR! stack     at PythonFinder.runChecks (C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\lib\find-python.js:136:21)
gyp ERR! stack     at PythonFinder.<anonymous> (C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\lib\find-python.js:225:16)
gyp ERR! stack     at PythonFinder.execFileCallback (C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\lib\find-python.js:271:16)
gyp ERR! stack     at exithandler (child_process.js:302:5)
gyp ERR! stack     at ChildProcess.errorhandler (child_process.js:314:5)
gyp ERR! stack     at ChildProcess.emit (events.js:210:5)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:270:12)
gyp ERR! stack     at onErrorNT (internal/child_process.js:456:16)
gyp ERR! stack     at processTicksAndRejections (internal/process/task_queues.js:80:21)
gyp ERR! System Windows_NT 10.0.18363
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\USERNAME\\AppData\\Roaming\\nvm\\v12.13.1\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd C:\A0\yarle-master\node_modules\node-expat
gyp ERR! node -v v12.13.1
gyp ERR! node-gyp -v v5.0.5
gyp ERR! not ok
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\USERNAME\AppData\Roaming\npm-cache\_logs\2020-11-15T09_16_35_916Z-debug.log

Any extra kind of formats in Evernote are still not tested.(I did not use them at all).

This might be out of scope, but it might be interesting: OCR PDFs (maybe through OCRmyPDF? Is there something more native to JS/TS?) that are included as attachments. Evernote describes their image recognition here:

If you export a note containing a PDF that has been processed by the OCR system, there will be two nodes in the document: data and alternate-data . The data node contains a base–64 encoded version of the original PDF and the alternative-data represents the searchable version of the same PDF.

In the DevonThink Forums one person from that company said that the OCR-Data is not available, but the above statement makes it sound it would be included.

internal/modules/cjs/loader.js:638
throw err;
^

Error: Cannot find module 'xml-stream'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (C:\Users\Sophia\dist\yarle.js:4:19)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: node --max-old-space-size=1024 ./dist/dropTheRope.js "--enexSource=GeneralNotes.enex" "--outputDir=./out" "--include-metadata" "--zettelkasten" "--plaintext-notes-only" "--outputFormat=ObsidianMD"
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\Sophia\AppData\Roaming\npm-cache_logs\2020-08-15T12_18_58_698Z-debug.log

Getting this error message on Windows:

C:\A0\yarle-master>npm i

> [email protected] install C:\A0\yarle-master\node_modules\node-expat
> node-gyp rebuild


C:\A0\yarle-master\node_modules\node-expat>if not defined npm_config_node_gyp (node "C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
gyp ERR! find VS
gyp ERR! find VS msvs_version not set from command line or npm config
gyp ERR! find VS VCINSTALLDIR not set, not running in VS Command Prompt
gyp ERR! find VS could not use PowerShell to find Visual Studio 2017 or newer
gyp ERR! find VS looking for Visual Studio 2015
gyp ERR! find VS - not found
gyp ERR! find VS not looking for VS2013 as it is only supported up to Node.js 8
gyp ERR! find VS
gyp ERR! find VS **************************************************************
gyp ERR! find VS You need to install the latest version of Visual Studio
gyp ERR! find VS including the "Desktop development with C++" workload.
gyp ERR! find VS For more information consult the documentation at:
gyp ERR! find VS https://github.com/nodejs/node-gyp#on-windows
gyp ERR! find VS **************************************************************
gyp ERR! find VS
gyp ERR! configure error
gyp ERR! stack Error: Could not find any Visual Studio installation to use
gyp ERR! stack     at VisualStudioFinder.fail (C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:121:47)
gyp ERR! stack     at C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:74:16
gyp ERR! stack     at VisualStudioFinder.findVisualStudio2013 (C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:351:14)
gyp ERR! stack     at C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:70:14
gyp ERR! stack     at C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:372:16
gyp ERR! stack     at C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\lib\util.js:54:7
gyp ERR! stack     at C:\Users\USERNAME\AppData\Roaming\nvm\v12.13.1\node_modules\npm\node_modules\node-gyp\lib\util.js:33:16
gyp ERR! stack     at ChildProcess.exithandler (child_process.js:302:5)
gyp ERR! stack     at ChildProcess.emit (events.js:210:5)
gyp ERR! stack     at maybeClose (internal/child_process.js:1021:16)
gyp ERR! System Windows_NT 10.0.18363
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\USERNAME\\AppData\\Roaming\\nvm\\v12.13.1\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd C:\A0\yarle-master\node_modules\node-expat
gyp ERR! node -v v12.13.1
gyp ERR! node-gyp -v v5.0.5
gyp ERR! not ok
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\USERNAME\AppData\Roaming\npm-cache\_logs\2020-11-15T09_29_06_676Z-debug.log

C:\A0\yarle-master>

Currently, the Zettelkasten IDs are generated based on the creation time of the note. A smarter way would be to generate them to represent a logical hierarchy of the notes driven by folder hierarchy (in where the enex files are placed).

Example:
Let's assume that we have 3 notebooks in Evernote: Goals with notes titled g1, g2, DailyNotes with notes titled dn1, dn2 and dn3, and Projects with notes titled project1 and project2.
Goals and DailyNotes are in the same NotebookStack called Journal.

Let's assume that the working dir is Notes. Put projects.enex to it and create a folder called Journal. Then put goals.enex and dailynotes.enex to Journal. Hence we created a folder-representation of Notebook Stacks of Evernote.

The smart Zettelkasten Ids would be generated as follows:

Journal -> 1

dailynotes -> 1.1

dn1 -> 1.1.1
dn2 -> 1.1.2
dn3 -> 1.1.3

goals -> 1.2

g1 -> 1.2.1
g2 -> 1.2.2

project.enex -> 2

project1 -> 2.1
project2 -> 2.2

Hi!
I have no idea where to find an Enex file
I only can see a notes.exb file inside C:\Users\Polaris\Evernote\Databases

any idea?
Thanks

Related: #90

From the turndown readme:

Turndown uses backslashes (\) to escape Markdown characters in the HTML input. This ensures that these characters are not interpreted as Markdown when the output is compiled back to HTML. For example, the contents of <h1>1. Hello world</h1> needs to be escaped to 1\. Hello world, otherwise it will be interpreted as a list item rather than a heading.

I happen to be one EN user who has written markdown symbols into his notes, since inline code and block quotes were not part of the editor.

I think we should provide an option to the user to not escape symbols that look like markdown at all. I think it should be optional, since the point turndown makes is a valid one. But not escaping repairs internal links with underscores etc. immediately. It also doesn't escape > symbols for blockquotes and generally does the right thing - in my case.

A simplistic solution would be to simply do this, when the specific option is set, in turndown-service:

TurndownService.prototype.escape = ((str: string) => str);

And maybe that's enough as a first step. However we might want to instead overwrite the TurndownService.prototype.process function and add here those types of elements where we don't want to escape MD symbols:

...
if (node.nodeType === 3) {
      replacement = node.isCode ? node.nodeValue : self.escape(node.nodeValue)
 } ...

It would be relatively easy to add here other nodes (like divs that are not wrapped by a header tag) that also shouldn't be escaped.

As far as I see, currently Yarle supports the following metadata: tags, created, updated and location.

Please also support the source URL that is part of the metadata in Evernote.

To be able to import notes easier, the tool should separate simple (just md) notes from the more complex ones (notes with external resources).

Many unit tests rely that the "Create At", "Updated At" and zettelkasten IDs will be rendered according to the user's timezone, which may differ while running unit tests.

Tests could use a library like timezone-mock to ensure they always run on a given timezone and therefore always produce the same expected output.

Given a note that looks like this in Evernote:

Note export looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE en-export SYSTEM "http://xml.evernote.com/pub/evernote-export3.dtd">
<en-export export-date="20201010T165755Z" application="Evernote Legacy" version="Evernote Mac 7.14.1 (458325)">
    <note>
        <title>ListTest</title>
        <content><![CDATA[<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd"><en-note><ul><li><div>Level1</div></li><ul><li><div>Level2</div></li><ul><li><div>Level3</div></li></ul></ul></ul><div><br /></div></en-note>]]></content>
        <created>20201010T165715Z</created>
        <updated>20201010T165734Z</updated>
        <note-attributes>
            <latitude>60.5566790535642</latitude>
            <longitude>24.98012759429327</longitude>
            <altitude>67.66667938232422</altitude>
            <author>Martin Hähnel</author>
            <source>desktop.mac</source>
            <reminder-order>0</reminder-order>
        </note-attributes>
    </note>
</en-export>

The note itself here as a more nicely rendered version:

<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd">
<en-note
  ><ul>
    <li><div>Level1</div></li>
    <ul>
      <li><div>Level2</div></li>
      <ul>
        <li><div>Level3</div></li>
      </ul>
    </ul>
  </ul>
  <div><br /></div
></en-note>

After running a command like this:

npm run start -- --enexSource=./ListNote.enex --outputDir=./out/ --includeMetadata

We get a markdown file that looks like this:

# ListTest
*   Level1
    

*   Level2
    

*   Level3

What it should have been instead:

# ListTest
*   Level1
    *   Level2
        *   Level3

Beside of getting Markdown files, it would be nice if an index file, e.g. a file that contains references to the notes, would be generated per Notebook.

In order to eliminate problems and difficulties around the installation of yarle and its 3rd party dependencies, it would be great to provide the application as a Docker image instead.

Hey Akos,

Very minor bug here but thought I should flag it anyway.

Ran out of disk space while doing the conversion.

Failed to convert note: # Golgi
 Error: ENOSPC: no space left on device, open '/Users/natske/Applications/yarle/./out/simpleNotes/My Notes2/201107121226 golgi.md'
    at Object.openSync (fs.js:462:3)
    at Object.writeFileSync (fs.js:1362:35)
    at Object.exports.writeMdFile (/Users/natske/Applications/yarle/dist/utils/file-utils.js:7:12)
    at Object.exports.processNode (/Users/natske/Applications/yarle/dist/process-node.js:30:22)
    at EventEmitter.<anonymous> (/Users/natske/Applications/yarle/dist/yarle.js:31:28)
    at EventEmitter.emit (events.js:315:20)
    at SAXStream.<anonymous> (/Users/natske/Applications/yarle/node_modules/xml-flow/lib/xml-flow.js:142:15)
    at SAXStream.emit (events.js:315:20)
    at SAXParser.me._parser.<computed> [as onclosetag] (/Users/natske/Applications/yarle/node_modules/sax/lib/sax.js:258:17)
    at emit (/Users/natske/Applications/yarle/node_modules/sax/lib/sax.js:624:35) {
  errno: -28,
  syscall: 'open',
  code: 'ENOSPC',
  path: '/Users/natske/Applications/yarle/./out/simpleNotes/My Notes2/201107121226 golgi.md'
}
Note # Golgi
 converted successfully.
Notes processed: 3618
Conversion finished: 3618 succeeded, 0 failed

Note that in the final statement it says that '0 failed' even though many hundreds didn't write because there was no disk space available.

It might be worth it to consider implementing the use of a config file which can be used instead of cli parameters to control yarle. I could see a yarle.json that we could use specify input/output folders, outputformat and so on.

At the moment we don't have a good way to deal with notes like this:

This generates enml like this (only the note part here):

<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd">
<en-note>
  <ul>
    <ul>
      <li>
        <div>Level 2</div>
      </li>
    </ul>
  </ul>
  <div>
    <br />
  </div>
</en-note>

In theory we should get Markdown that looks like this (note the 8 spaces that signify a level 2 list item):

# test - sublists - double indent

        *   Level 2

But we get instead:

# test - sublists - double indent

*   Level 2

At the moment this is unavoidable because the sublist fix (see here) relies on having a sibling <li> next to the sublist <ul>/<ol>.

In order to make these kinds of lists work we would need to overwrite the turndown rule for list items, I think.

Links with a file:///... URL should not be converted to [[file:///...]] in Obsidian format, but to ordinary Markdown links like http://... links.

ENML that includes a code block like this:

<div style="box-sizing: border-box; padding: 8px; font-family: Monaco, Menlo, Consolas, &quot;Courier New&quot;, monospace; font-size: 12px; color: rgb(51, 51, 51); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background-color: rgb(251, 250, 248); border: 1px solid rgba(0, 0, 0, 0.15);-en-codeblock:true;">
    <div># Add a new remote upstream repository</div>
    <div>git remote add upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git</div>
    <div>
      <div>
        <br />
      </div>
      <div>
        <br />
      </div>
    </div>
    <div># Sync your fork</div>
    <div>git fetch upstream</div>
    <div>git switch master</div>
    <div>git merge upstream/master</div>
  </div>

Is not converted correctly into a fenced code block. Instead it is handled as if it would be just normal Text:

\# Add a new remote upstream repository
git remote add upstream https://github.com/ORIGINAL\_OWNER/ORIGINAL\_REPOSITORY.git

\# Sync your fork
git fetch upstream
git switch master
git merge upstream/master

There should be a rule that detects these code blocks, probably based on the -en-codeblock:true part of the style attribute.

Even the best conversion to Markdown can never be lossless. Some notes may look ugly or become less readable after conversion, or parts of the text may be lost during conversion for some reason.

To be 100% sure that nothing is lost, and to be able to view the original formatting even without a Evernote client, it would be nice if Yarle could (optionally) add a link to the original HTML for every Markdown note. The original HTML files could be stored in a separate subdirectory, similar to the attachments. This directory could be either created from the enex file by Yarle. But Evernote can also export the notes in a directory with individual HTML files. Yarle will then only need to add the links to the files in this directory.

Attatch file is enex file with only one note, it has 4 pictures, after convert to markdown, its md file's four picture point to same picture, though its _rescources file still have 4 pictures.

Most note convert OK, I just pick one convert wrong note in attached enex file.

My Command:
$ npm run start -- --enexSource=/mnt/c/temp/onenote_exp/Evernote_abc.enex --outputDir=./out_food_abc --include-metadata --zettelkasten --plaintext-notes-only --outputFormat=StandardMD
yarle_bug.zip

This issue is thought as an umbrella issue to think about how a more robust linking mechanism could look like.

At the moment the conversion of internal links is supposed to work like this:

• If there is an internal link in the enml, then this gets converted to a WikiLink that links to the title of the note that is linked.

This is a good general approach however it has its problems:

1. Titles that are too long, contain spaces or other symbols are not converted correctly
   • Filenames get lowercased, however, internal links are not: #73
   • Symbols sometimes get escaped: #90
   • Spaces get converted to underscores: #91
   • Too long titles are truncated when new files are created, but not when linking titles: #91
2. Notes cannot contain certain symbols to work with certain Markdown apps (e. g. Obsidian):
   • These are (at least)* " \ / < > : |
3. Not everybody is using WikiLinks:
   • It should be possible to use MarkdownLinks for internal Links as well.

Proposed solution

As far as I can see it, most importantly we need to make sure that the internal link === filename.

This means we need to normalize internal Links and filenames in the same way. I think we should write a function that - given a string (the title of the note) - returns a "safe" version of that string and it is used in the file creation as well as in the internal link creation steps.

Furthermore I think it should be possible for the user to decide if internal links should be in Markdown or Wiki format.

Finally, inside internal links the link should also contain the (unescaped) title of the note as the name of the link. E. g.:

[[nameOfFile|Display Name]]
[Display Name](nameOfFile)

Maybe the last part should be optional.

This solution would solve all the corner cases we are aware of right now and makes sure that there is only one place to add to/change if new cases come up: the normalizeTitle function.

Feedback and comments are appreciated!

Internal links added in Evernote cannot be supported, because the ids (which are used as links) are not embedded in the enex file.

It seems that newer versions of Evernote split the notebooks in chunks when they are larger than 100MB, like Notebook.enex, Notebook [2].enex, Notebook [3].enex, ... It would be nice if Yarle recognizes when there are "follow-up" notebooks and imports their contents as if they belonged to the first notebook file.

With the most recent source (as of this post), certain notes in enex format will fail to export. I have attached an example note, which produces the following output:

chris yarle-master $ npm run start -- --enexFile=ShadowWork.enex --outputDir=./ShadowWork

> [email protected] start /home/chris/yarle-master
> node ./dist/dropTheRope.js "--enexFile=ShadowWork.enex" "--outputDir=./ShadowWork"

/home/chris/yarle-master/dist/dropTheRope.js:21
    if (options.enexFile.endsWith('.enex')) {
                         ^

TypeError: Cannot read property 'endsWith' of undefined
    at Object.exports.run (/home/chris/yarle-master/dist/dropTheRope.js:21:26)
    at Object.<anonymous> (/home/chris/yarle-master/dist/dropTheRope.js:38:9)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node ./dist/dropTheRope.js "--enexFile=ShadowWork.enex" "--outputDir=./ShadowWork"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/chris/.npm/_logs/2020-05-26T16_40_58_055Z-debug.log

I will attach the debug log as well.

I had actually intended to report a different error, but upon updating my source I started getting this error. Once this is fixed, I suspect you will encounter a second problem exporting this note. The output of that error, from older source, was:

chris yarle-master $ npm run start -- --enexFile=ShadowWork.enex --outputDir=./ShadowWork

> [email protected] start /home/chris/yarle-master
> node ./dist/dropTheRope.js "--enexFile=ShadowWork.enex" "--outputDir=./ShadowWork"

/home/chris/yarle-master/dist/process-resources.js:29
    const src = `./_resources/${relativeResourceWorkDir}/${resourceHashes[hash].replace(/ /g, '\ ')}`;
                                                                                ^

TypeError: Cannot read property 'replace' of undefined
    at addMediaReference (/home/chris/yarle-master/dist/process-resources.js:29:81)
    at Object.exports.processResources (/home/chris/yarle-master/dist/process-resources.js:24:26)
    at Object.exports.processNode (/home/chris/yarle-master/dist/process-node.js:19:39)
    at Object.exports.dropTheRope (/home/chris/yarle-master/dist/yarle.js:35:28)
    at Object.exports.run (/home/chris/yarle-master/dist/dropTheRope.js:20:11)
    at Object.<anonymous> (/home/chris/yarle-master/dist/dropTheRope.js:23:9)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node ./dist/dropTheRope.js "--enexFile=ShadowWork.enex" "--outputDir=./ShadowWork"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/chris/.npm/_logs/2020-05-26T16_08_04_816Z-debug.log

2020-05-26T16_40_58_055Z-debug.log

ShadowWork.enex.zip