NotDeft

Table of Contents

NotDeft is an experimental variant of Deft, an “Emacs mode for quickly browsing, filtering, and editing directories of plain text notes.” This variant is somewhat less configurable than mainline Deft, but (like the Notmuch Emacs mode for managing email) it features a Xapian backend for efficient free-text search over potentially very large numbers of note files.

The key differences of NotDeft are:

  1. Rather than supporting a single, customizable notdeft-directory (tree) of note files, it supports a customizable notdeft-path search path of directory trees.
  2. It supports optional invocation of a notdeft-xapian-program, which uses the Xapian library to implement free-text search across note files, with convenient query syntax. The search is performed across all directory trees on notdeft-path, and further narrowing down of the result set can then be done incrementally by typing in a search string for purposes of filtering (as in Deft).

1 Source Code

NotDeft source code repository can be found at
https://github.com/hasu/deft

2 NotDeft Installation

NotDeft can be installed either as a package, or manually.

2.1 Installation as a Package

To install as a package, first download the package via the URL http://terohasu.net/deft/notdeft-0.3.tar, and then install the TAR file with

M-x package-install-file

You can check whether the package has been installed by evaluating

(describe-package 'notdeft)

No documentation is shown by that command, but it does show the location of the package's files, allowing navigation to the documentation. Alternatively, one might implement a command for that purpose; for example:

(defun notdeft-open-readme ()
  (interactive)
  (find-file
   (expand-file-name
    "README.org"
    (package-desc-dir
     (cadr (assq 'notdeft package-alist))))))

While installing the package does unpack the Xapian backend source code, it does not build it or configure it; see Building the Xapian Backend and Configuring the Xapian Backend.

2.2 Manual Installation

Optionally, byte-compile the Emacs Lisp files with the command

make

Add the directory containing those files to the Emacs search path by adding

(add-to-list 'load-path "/path/to/notdeft/repository")

to your Emacs startup file (e.g., “~/.emacs”). Also add

(require 'notdeft-autoloads)

to the startup file, to cause NotDeft to be loaded when you execute

M-x notdeft

2.3 Building the Xapian Backend

To enable Xapian search queries, you should build the notdeft-xapian C++ program in the “xapian” directory. On some systems simply typing

make

should do the trick, provided that the required libraries have been installed. On other systems you will need to find the right C++ compiler incantation for building the program.

3 NotDeft Configuration

Once the notdeft feature has been loaded, you can see and edit all of its configuration options and their documentation with the command

M-x customize-group RET notdeft RET

The most essential settings are

notdeft-xapian-program
to specify the path of the Xapian search tool
notdeft-path
to specify the location(s) of your notes

If your notes are not in a fixed directory, but you'd rather discover the directories programmatically, it may be convenient to set notdeft-path in your startup file. For example:

(setq notdeft-path `("~/notes" ,@(file-expand-wildcards "~/*/notes")))

3.1 Org Mode Integration

NotDeft is somewhat specialized for managing notes in the Org format. If you do use Org mode for editing your notes, and your Org mode is version 9, you may want to include

(require 'notdeft-org9)

in your Org startup code, to set up support for “deft:” and “notdeft:” links in org-mode. A “deft:” link names a note by its non-directory filename, whereas a “notdeft:” link names a NotDeft Xapian search.

Org mode's org-store-link command may be used to capture any Xapian search in a NotDeft buffer, to be later inserted with org-insert-link. The notdeft-org9 feature also defines NotDeft-specific notdeft-kill-ring-save-org-link and notdeft-insert-org-link commands for capturing and inserting “deft:” links.

3.2 Note File Format

The default is to have the note filename notdeft-extension set to "org" to indicate the Org format. If you prefer some other note format, you should change that setting, which can be done with

M-x customize-variable RET notdeft-extension RET

The configured notdeft-extension is used by default when creating new notes, but a note collection can also use other extensions. There are none by default, but you can define such secondary extensions with

M-x customize-variable RET notdeft-secondary-extensions RET

For example, one might set these as

(setq notdeft-extension "txt")
(setq notdeft-secondary-extensions '("md" "org" "scrbl"))

3.3 Configuring the Xapian Backend

To have NotDeft use the program, you will have to specify its absolute path in the notdeft-xapian-program variable. You could use M-x customize-variable to set it, or simply

(setq notdeft-xapian-program "/path/to/notdeft-xapian")

If you installed as a package, and built the notdeft-xapian executable in that location, then the appropriate setting may be

(setq notdeft-xapian-program
      (expand-file-name
       "xapian/notdeft-xapian"
       (package-desc-dir
        (cadr (assq 'notdeft package-alist)))))

Such code must appear after

(package-initialize)

See the other notdeft-xapian-* customization variables for configuring the Xapian indexing and searching behavior. Most notably:

  • The configuration variable notdeft-xapian-max-results controls the maximum number of files to list in a notdeft-mode buffer. You may set it to 0 to always have all results displayed.
  • The default is to order the results so that most recently edited files are listed first, but you may change this behavior by setting notdeft-xapian-order-by-time to nil, in which case Xapian's ranking mechanism is used instead.

4 Search Query Syntax

Press Tab in a *NotDeft* buffer to enter a search query. The usual Xapian search query syntax is available. Operators such as AND, OR, and XOR are available, and they may also be written in lowercase (or mixed case) if notdeft-xapian-boolean-any-case is set to t. The NOT operator is also available if notdeft-xapian-pure-not is t. It is possible to query for a phrase by quoting the phrase (e.g., "Deft for Emacs"). To look for a search term without stemming, give it capitalized (e.g., "Abstract" will not match “abstraction”). Wildcards in search terms are not supported (trailing wildcards are supported by Xapian, but not enabled in NotDeft).

4.1 Prefixes

The following prefixes are supported by NotDeft:

file:
Indicates that the search term must appear in the (non-directory, non-extension) filename.
title:
Indicates that the search term must appear in the title.
  • Title is specified either as the first non-empty non-comment line, or as the file property (or Org mode “in-buffer setting”) #+TITLE. (Multiple #+TITLE lines are not supported.)
tag:
Indicates that the search term must appear among the tags given to the document.
  • The tags for a note are specified either with the standard Org file property #+FILETAGS, or the custom file property #+KEYWORDS.

4.2 Query Modifiers

The following custom query syntax is supported:

!all
Prefix a query with !all to show all matching results.
!time and !rank
Prefix a query with !time to have results sorted by file modification time, or with !rank to have them sorted by relevance, regardless of the notdeft-xapian-order-by-time setting.
!file
Prefix a query with !file to have results sorted by (non-directory) file name.

A space character must be used to separate the above keywords from the rest of the query string.

The !file modifier might be useful for instance when you have file names such as “2017-01-01-0001.JPG” and “2017-09-19-0123.JPG”, and you would like to see them in chronological order by “creation time”, even if some of the files have been edited, and consequently have had their modification times changed.

5 Example Notes

No special markup is necessarily required:

this is a title

This is body text.

Comments can be included, and they are ignored when searching:

# this is a comment
this is a title

This is body text.

Org mode's #+TITLE syntax is supported:

# this is a comment
#+TITLE: this is a title
# this is a comment

This is body text.

A note can be tagged, e.g., with the tags “some” and “tags”:

#+TITLE: this is a title
#+KEYWORDS: some tags

This is body text.

Instead of the #+KEYWORDS syntax, we can use the Org standard #+FILETAGS syntax:

#+FILETAGS: :some:tags:
this is a title

This is body text.

Stemming is used also on tags, and so the query “tag:tag” will find these two notes (assuming English stemming—see notdeft-xapian-language).

6 Example Search Queries

It is simple to find all notes containing both the words Emacs and Org:

Emacs AND Org

If you have a lot of notes about Org mode, and few about other Emacs matters, it may be interesting to use

Emacs AND NOT Org

which works if the notdeft-xapian-pure-not option is set.

While you're often likely to be more interested in recent (or best maintained) notes, sorting by relevance can be useful particularly when there are multiple search terms: you may be more interested in seeing notes that contain all the terms instead of just one of them. You may use “!rank” to enable relevance-based ranking for a specific query:

!rank Emacs Org Deft

If, on the other hand, you use a single, common search term, and have a lot of documents, you may run into your notdeft-xapian-max-results limit, and miss out on some documents. In this case, you might use

!all Emacs

to list all documents mentioning Emacs.

If, unlike in the above case, you just want to see all documents that are about Emacs specifically, you may get more useful results with the query

title:Emacs

to only find documents whose title indicates that they concern Emacs. Or, to be more thorough, you might want to make sure you also find notes with the word Emacs in the filename:

title:Emacs OR file:Emacs

You can combine prefixes and “bracketed subexpressions”:

title:(Ayn AND Rand)

which will match both “Ayn Rand” and “Rand, Ayn” in a title.

Phrase searches are not expected to be useful for tags, and hence the query

tag:"some tags"

will not yield any results, regardless of the sets of tags in your notes, or the way they have been declared.

7 Transient Directories

It is acceptable for the notdeft-path to contain transient directories, as any non-existing directories are simply ignored; if and when they re-appear, they can be included in subsequent searches by first refreshing the searched notdeft-directories with the notdeft-refresh command (or C-c C-g in *NotDeft*).

Note, however, that notdeft-path is normally set only once, and if you use wildcards, it may not contain directories that get mounted or copied over later. Thus, if your Emacs startup file says

(setq notdeft-path `("~/notes" ,@(file-expand-wildcards "~/*/notes")))

and the directory

~/phone/notes

becomes available after Emacs has already started, your notdeft-path may not include the newly available directory.

To make it more convenient to deal with directories that appear after Emacs startup, you are allowed to include restricted code forms in your notdeft-path definition:

(setq notdeft-path '("~/notes" (file-expand-wildcards "~/*/notes")))

Now, whenever you notdeft-refresh, your recomputation of notdeft-directories will also re-evaluate any such code forms.

8 Invoking NotDeft from Another Mode

To quickly find relevant notes when in another buffer, you might use

M-x notdeft-open-query

which then interactively asks for a search query for opening up in a NotDeft buffer. That command can of course be bound onto a key.

You might also implement additional commands in terms of the above, for example for quickly listing documents tagged in a certain way:

(defun my-open-todo-notes ()
  (interactive)
  (notdeft-open-query "tag:todo"))

A command similar to notdeft-open-query is

M-x notdeft-lucky-find-file

which also asks for a search query, but then proceeds to open up the most highly ranked result file directly, without going via a *NotDeft* buffer. This command is similar to find-file in Emacs, but avoids having to specify the path of the file you're interested in; instead, this approach to “file finding” relies on sufficiently unique titling or tagging of the notes involved.

The notdeft-rename-file command can be useful for renaming a note file that was perhaps created without a proper name (e.g., by using C-c C-n). Having written a note in a current buffer, issue the command

C-u M-x notdeft-rename-file

to enter a new basename for the file of that buffer. The C-u prefix causes the default value to be derived from the title of the note, as extracted from the buffer contents. The same command also works in a *NotDeft* buffer, affecting the currently selected file.

The NotDeft commands that are usable from outside notdeft-mode might be bound to key combinations for convenient access. To facilitate this, NotDeft provides a notdeft-global feature, which exports a keymap for those commands. That keymap can be bound to a prefix key. For example:

(require 'notdeft-global)
(global-set-key [f6] 'notdeft-global-map)

after which the command [f6] o should invoke the notdeft-open-query command in any mode that does not override the binding for F6 with something else.

9 Quick Note Capture

To quickly create a new note file from any buffer, you can use

M-x notdeft-new-file

That command is also bound to C-n in notdeft-global-map, and if that keymap is bound to the prefix [f6], for example, then you can create a new note with the key combination [f6] C-n.

Org mode has its own “capture” mechanism, and you can certainly configure capturing into a file that resides in a NotDeft directory. For example:

(setq org-directory "~/notes") ;; default Org files location
(setq notdeft-path (list org-directory)) ;; NotDeft search path
(setq org-default-notes-file (concat org-directory "/notes.org"))
(global-set-key [f7] 'org-capture)

which defines "~/notes" as the sole NotDeft directory, and has the key F7 initiate an org-capture, by default into the file "~/notes/notes.org". After completing capture, you can go back to the previously captured item with

C-u C-u M-x org-capture

The capture facility supports the definition and use of org-capture-templates for different purposes.

A caveat with Org capturing is that unless you have already opened the capture file under NotDeft, any newly captured items may not immediately get noticed by NotDeft. To ensure that NotDeft is aware of any changes, one could write custom commands which register the capture file, for example with

(notdeft-register-file org-default-notes-file)

Note that different org-capture-templates may define different capture locations.

10 Adding Attachments to Notes

NotDeft has a simple mechanism to support “attaching” files to notes, one that is agnostic to the note file format. If you have a note file

~/notes/deft-for-emacs.txt

you can use the command C-c S to move the file into a subdirectory of the same name, so that the file's pathname becomes

~/notes/deft-for-emacs/deft-for-emacs.txt

Now you can copy/move/link any attachments for the note into that subdirectory, and it is convenient to move the note together with its attachments using a regular file manager.

To move a note from within *NotDeft*, the command C-u C-c m can be used to move it under another NotDeft root directory, where the prefix C-u causes it to be moved together with its subdirectory.

When the attachments reside in the same directory as the note itself, in Org mode it is then easy to add a “file:” link to any attachment with the command C-u C-c C-l. For example, if the attachment directory contains a file named “2017-01-01-0001.JPG”, then a “file:” link to it would be simply

[[file:2017-01-01-0001.JPG]]

and the command C-c C-x C-v can be used to toggle inline display of images.

Org itself has its own attachment management mechanism, whose action menu is bound to C-c C-a. This mechanism allows an attachment directory to be associated with an Org heading (as identified by information stored within the heading's properties), and thus the NotDeft note file itself can reside directly within a NotDeft root directory. Org has no command for moving an Org file together with its attachments, however.

To make the Org mechanism compatible with the NotDeft mechanism, one can store the attachments in the same (sub)directory as the note file itself, by specifying that directory with the ATTACH_DIR property. For example:

* Bergen, Norway                              :ATTACH:
  :PROPERTIES:
  :ATTACH_DIR: ./
  :Attachments: 2017-01-01-0001.JPG 2017-09-19-0123.JPG
  :END:

This way it is still convenient to move a note together with its attachments, and Org commands such as C-c C-a o (for opening the attachments) can still be used.

11 Note Archival

To archive away a note so that its contents will no longer be included in a search, one can press C-c C-a from within *NotDeft*. This is a note format agnostic archival method, as the entire note file gets moved into a notdeft-archive-directory, with the default name of

"_archive"

meaning that a note file whose original path is

~/notes/deft-for-emacs.txt

would get moved to

~/notes/_archive/deft-for-emacs.txt

Any directories whose names begin with an underscore will be excluded from Xapian searches, and thus such an archived note will no longer clutter search results.

In Org mode one can use Org's own archival mechanism to archive just a part of a note document subtree, and the archival file will also be excluded from Xapian searches, provided that its filename extension is not notdeft-extension or one of the notdeft-secondary-extensions. Org's default extension is

org_archive

which by default is not an extension recognized by NotDeft.

12 Capturing Data from External Applications

The org-protocol feature of Org mode provides a way for some external applications to interface with Emacs and Org, and the same mechanism can also be adapted for capturing data into NotDeft. For example, data can be sent from Firefox to NotDeft using the predefined store-link and capture protocols.

The mechanism works by the external application invoking emacsclient, and for this to work you should have an Emacs server running in the Emacs instance you want to use to receive data into NotDeft. A server can be started by evaluating

(server-start)

12.1 org-protocol Content Type in Firefox

To configure Firefox to support the org-protocol: scheme, first open about:config, and add a boolean property

network.protocol-handler.expose.org-protocol false

Then craft an HTML file such as

<html>
  <body>
    <a href="org-protocol://store-link?url=URL&title=TITLE">link</a>
  </body>
</html>

and open that file in Firefox, and click the link, after which a “Launch Application” dialog is presented. “Choose other Application”, tick the box “Remember my choice for org-protocol links”, and specify emacsclient as the executable.

That application selection can later be modified from Firefox “Preferences” / “Applications”. If required, the “Content Type” should be removable at least by editing the “mimeTypes.rdf” file in the Firefox profile.

12.2 store-link from Firefox

There is nothing NotDeft specific about the store-link Org protocol, as it merely stores a link to the Emacs kill-ring for yanking. To configure Firefox to support the protocol, just add a suitable bookmarklet (e.g., to the “Bookmarks Toolbar”). The bookmark “Location” can be specified as

javascript:location.href='org-protocol://store-link?url='+encodeURIComponent(document.location)+'&title='+encodeURIComponent(document.title);void(0);

By selecting that bookmark a link to the current page can be inserted in Emacs with C-y, or with

M-x org-insert-link

which is bound to C-c C-l in Org.

12.3 capture from Firefox

Configuring the capture protocol for use with NotDeft is slightly more involved, if we assume that we want to choose a filename based on the page title, and if we also want to open the note file (associated with that title) in NotDeft (so that NotDeft also detects the changes to the file).

The Firefox bookmarklet can for example be

javascript:location.href='org-protocol://capture?template=w&url='+encodeURIComponent(document.location)+'&title='+encodeURIComponent(document.title)+'&body='+encodeURIComponent(window.getSelection());void(0);

which now also sends any currently selected text over to Emacs.

Now we must also define the “w” template as one of the org-capture-templates, and the definition can be

(require 'org-protocol)
(require 'notdeft-autoloads)

(setq org-capture-templates
      '(("w" "capture selection into NotDeft" plain
         (file (lambda ()
                 (notdeft-switch-to-file-named
                   (plist-get org-store-link-plist :description))))
         "%l\non %u\n\n%i"
         :empty-lines-before 1)))

This definition assumes that the :description is available from org-store-link-plist, and that it corresponds to the document.title; this may be undocumented functionality, but works in Org mode 9.1.1.

13 See Also

The file “notdeft.el” has some more documentation.


Tero Hasu