Show HN: Apple Notes Liberator – Extract Notes.app Data and Save It as JSON

oneearedrabbit

I am excited to see more solutions that work with Apple Notes database directly. This is my go-to application to capture unstructured memos while researching various topics. Last year I got carried away and built a simple site generator for Notes [0]. As a fun exercise I implemented ORM models to query Sqlite database in Ruby:

  User::Account.server(database: "tmp/test.sqlite").first.
    folders_dataset.offset(1).first
    notes.
    map(&:attachments).flatten.
    find { |el| el.is_a?(User::ObjectTypes::EmbeddedPublicJpeg) &&
      el.height > 700 &&
      el.summary[/animal/i] }.
    media_filepath
 => "Accounts/<uuid>/Media/F669B83A-0C47-408E-9645-015737F3B11F/Pasted Graphic.jpg"

Apple Notes' database structure is a fascinating product story. Notes started as a simple note-taking application that stored its content as raw HTML in a few simple tables. However, as the product grew and needed more features like collaboration, they had to make changes and add new columns and tables. They also implemented two flavors of CRDT-like algorithms that evolved over time: one for plaintext chunks and the other one for embedded objects like tables. On top of that, every change that Apple makes must be backward compatible to support legacy devices and systems; but backward-compatibility was not always possible and users had to go through “Migrate your Notes” steps, but it is again a product story.

I wish that one day, someone from the development team would write an article about it.

[0] https://devlog.notespub.com/2022/08/site-generator-for-apple...

simonw

I wrote a similar tool in Python a few weeks ago, which uses AppleScript to liberate the data and saves it to a SQLite database: https://datasette.io/tools/apple-notes-to-sqlite

I didn't actually know AppleScript before writing this tool... but it turned out ChatGPT knew it well enough to unblock me and provide me with exactly the code I needed to build the rest of the project! https://til.simonwillison.net/gpt3/chatgpt-applescript

Apple Notes Liberator creates a copy of the SQLite database and then runs queries against that directly to extract the data.

I chose not to do that (despite being all-in on SQLite for everything) because I worry about future changes to the software baking my script - I figured the AppleScript interface was less likely to have breaking changes in the future.

ecliptik

A feature that a lot of Apple apps have that doesn't seem talked about much is they can use standard protocols and not require iCloud.

Mail.app/Reminders.app/Notes.app can use IMAP, Cal.app with CalDAV, and Contacts.app with CardDAV [1].

It doesn't offer a lot of the features iCloud has like advanced formatting or collaboration, but is a useful for simple standards based syncing and portability.

I've done this with my iPhone/Macs and Fastmail for the last few years and it works great [2].

1. https://support.apple.com/guide/iphone/set-up-mail-contacts-...

2. https://www.fastmail.help/hc/en-us/articles/1500000279941

samwillis

Super interesting, particularly the link through to the blog with all the research into how Notes works.

A particularly interesting thing that I don't see mentioned anywhere is that Apple Notes uses CRDTs (conflict free replicated data types) to enable both shared notes and multiple concurrent edits. With the CRDTs resolving the conflicts.

It looks like the info on the page may be a good place to start looking to how that works: https://www.ciofecaforensics.com/2020/10/20/apple-notes-clou...

ChintanGhate

I have published an app [Exporter](https://apps.apple.com/us/app/exporter/id1099120373?mt=12) that does something similar - exports Apple Notes to HTML and markdown with the attachments. Give it a try, let me know if it solves your issue.

Also, why did you select JSON as the output format?

dangoor

Cool! Thanks for sharing. Also worth noting that there's a Mac app in the app store to export to markdown or html:

https://apps.apple.com/us/app/exporter/id1099120373?mt=12

(I haven't had a need yet, so I haven't tried it)

css

If anyone ever wanted to do something similar for iMessage data, I built a similar tool for that: https://github.com/ReagentX/imessage-exporter

mkmk

I consistently struggle with Notes and iMessage sync issues, and I suspect it’s due to being a customer for so many years and making my way through many devices over those years.

The UX of the Apple ecosystem is so nice but when it goes sideways you’re kind of out of luck, even if you have applecare.

I wish they had some sort of premium “fix my esoteric appleID account issues” service you could pay them for once a decade or so.

pjot

This is great! How do you account for updates to a note?

I’ve been building an archiver tool as well to convert notes into markdown and post to git using Apple Shortcuts.

The interop between almost all iOS data formats (notes, safari, calendar, etc.) the content graph[0] provides is really impressive.

[0]: https://support.apple.com/en-me/guide/shortcuts/apd4618db957...

vszakats

Here's a Python script that takes the JSON output from https://github.com/threeplanetssoftware/apple_cloud_notes_pa... (needs master), and rebuilds Notes in Markdown format (or HTML), with links, attachments, lists, etc.:

https://gist.github.com/vszakats/5a3bd939721d1dde6142d9ea3b2...

And a short, standalone JXA script, that outputs HTML, but loses some data, such as links: https://gist.github.com/vszakats/f24e7700428c1e694e20cee0b1c...

ericzawo

Love this. I am an Apple Notes absolutist (see: https://pbs.twimg.com/media/FY1rZvNWIAAIRzs.jpg). I have tried Evernote, Workflowy, Dynalist, Notion over the years for marrying personal / professional worlds but always (always!!) come back to Notes.

I have dreamed of the ability to have a Notes web app with custom CSS for years and a few other features.

Good luck with this project I'll be watching closely

dmd

I don't understand what value this tool adds over the library it uses under the hood (which includes a command line tool for exporting your notes; I've been using it for years). What is it for?

divbzero

Note that Notes.app can also sync with IMAP.

IMAP syncing supports basic formatting (bold, italic, underline) but not all features supported by iCloud syncing (attachments, checklists, photos).

rco8786

Oh neat. I’ve been using Notes extensively since the beginning of this year. Daily journaling. Personal TODO lists, grocery list (shared w my wife), and gym/workout tracking.

It works really well and has way more features than people realize. If you’re looking for like a “Notion-lite”, the answer might already be on your phone/MacBook.

Alifatisk

So now we know how to export data from the notes.app, messages.app & reminders.app.

What’s next?

munk-a

As a windows user, while we don't have an OS built-in equivalent, the standard for such tools (in the form of Sublime and the like) is just to operate with temporary buffers that get written out as files on user request. Sublime will write internal data to some persistent storage somewhere and be able to survive sudden crashes and restarts but when you'd like to save the data out to a fixed filepath it's pretty trivial to do so. I do find it quite weird that Apple has managed to cultivate a system where storage in separate files isn't taken as the default.

kepano

I am excited to see more ways to export data out of Apple Notes. Long term it would be amazing to see a solution that allows more interoperability with other apps.

Depending on what you are looking to achieve there are a few ways to export Apple Notes to Markdown files, but it's only a one-way out solution.

From the Obsidian forum: https://forum.obsidian.md/t/import-from-apple-notes-to-obsid...

shanecleveland

I have been adopting Apple Notes more after trying to use a lot of other note methods/apps. Always fall back to simple text in Notes. And I seem to be hearing of more people doing the same.

I recently set up a shortcut that works on both my Mac and iPhone to create a new note each day with the date as the title. And then each time I trigger the shortcut that day, it enters the time, location and weather. Easy way to keep a running "diary."

joe8756438

This is cool. I wanted to provide notes export for people using a note taking service I built and landed on a ios shortcut.

I think you can accomplish something close or the same? using shortcuts.

One obvious benefit is the interface is going to be more approachable than the jar to most people.

Shortcuts are also cool because you can build one and package it for others and even define user specific parameters that get filled on install.

rhettbull

Very nice reverse engineering work! I wrote a python library and CLI that does something similar but interacts with Notes via Scripting Bridge: https://github.com/RhetTbull/macnotesapp

reaperducer

I built something similar a while back.

But instead of pulling the data out of Notes, I took it from SMTP mail.

I don't know if it still works this way, but when I did it, if you used a non-iCloud email address to sync Notes, it saved the data as pain text in email messages that could be easily parsed.

podviaznikov

instead of extracting notes data as JSON my simple tool allows you to liberate your notes by publishing them to the web.

https://montaigne.io/

I use the same amazing library linked in the blog post.

akerr

Why not just use the built-in Shortcuts to export to your preferred format?

solalf

Awesome initiative! I wish Apple Notes would support Markdown. This is my go to app for so much of my life, à Notion style editor would be an absolute game changer.

mdmglr

"Liberator" implies it was locked down. But notes content has always been easily accessible with AppleScript.

jamesu

For a long while I just copied everything to an imap server and dumped the mailbox, but this looks like an even better solution.

kolinko

Nice! It throws an error that it cannot find the notes folder though.

Where is it located usually? Or what's the file name?

marcjensen

Thanks for creating this. I must say that I don't find things written in Java on HN very often.

yosito

It would be really cool to set this up to automatically sync notes to Obsidian.

frozencell

This but for the Mac Library now please (the format transfer are a mess)

egeozcan

I wish someone did this for Whatsapp without having to jailbreak my phone.

gudh

I wish there's tool to extract Apple Books' notes too