emacs/org/notdeft/web/notdeft-homepage.org

NotDeft (homepage)

NotDeft is an Emacs-based manager and local search engine for directories of plain text notes. NotDeft features a Xapian backend for efficient free-text search over potentially very large numbers of note files; in that respect it is like the Notmuch Emacs mode for managing email. NotDeft is a spin-off of the Deft note manager, and retains similar functionality for browsing, filtering, and managing note collections. While NotDeft inherits its user interface from Deft, that interface is used for managing search result sets of notes, rather than directory contents. When used together with Org mode and its support for document linking, NotDeft can also function as a “desktop wiki,” such that documents can also be found by following links, and not just by searching and filtering.

The NotDeft source code repository can be found at
https://github.com/hasu/notdeft

• Quick Start
• NotDeft's Deft Origins
• NotDeft Installation
  • Manual Installation from Source
  • Installation as a Package with straight.el
  • Installation from a Package File
  • Building the Xapian Backend
• NotDeft Configuration
  • Specifying Note File Locations
  • Choosing the Note File Format
  • Configuring the File Naming Convention
  • Configuring the Xapian Backend
• NotDeft Mode
  • Displaying Individual Filter String Matches
  • Using Multiple NotDeft Mode Buffers
  • Displaying File Path Information
• NotDeft Note Mode
  • Enabling NotDeft Note Mode based on Major Mode
  • Enabling NotDeft Note Mode Locally to a Directory
  • Enabling NotDeft Note Mode based on a Directory-Local Variable
• Using NotDeft from Non-NotDeft Modes
  • Access from NotDeft Note Buffers
  • Programmatic NotDeft Access
• NotDeft Note Syntax
  • Example Notes
• Search Query Syntax
  • Prefixes
  • Query Modifiers
  • Example Search Queries
• Command Popup Buffers
• Org Mode Integration
  • Using NotDeft and Org Mode as a Desktop Wiki Engine
• Quick Note Capture
• Adding Attachments to Notes
• Note Archival
• Capturing Data from External Applications
  • org-protocol Content Type in Firefox
  • store-link from Firefox
  • capture from Firefox
• Troubleshooting
  • When Search Queries Are Not Yielding Expected Results
• See Also

Quick Start

For the impatient, this section outlines one way of downloading, installing and setting up NotDeft.

Open a terminal, and cd into your home directory. Download the source code into some directory with

git clone https://github.com/hasu/notdeft.git

Then prepare and byte-compile Emacs Lisp files with the commands

cd notdeft
make

where make is assumed to invoke GNU Make.

Then build the Xapian backend by doing

cd xapian
make

If the make command fails, then you will need to ensure that you have the required libraries installed, and find the right C++ compiler incantation for building the notdeft-xapian program on your system. A notable library requirement for compiling the program is TCLAP.

To make NotDeft loadable in Emacs, add the following code to your Emacs startup file (e.g., “~/.emacs”):

  (add-to-list 'load-path "~/notdeft")
  (add-to-list 'load-path "~/notdeft/extras")
  (load "notdeft-example")

The “notdeft-example.el” file is a sample configuration for NotDeft, which sets up NotDeft for use with Org mode based note files, also enabling some optional components of NotDeft, and setting up some keybindings. It may be a useful starting point for a personal configuration.

Create a “~/.deft” directory, and copy some “.org” files there.

Launch Emacs with

emacs -f notdeft

Press TAB to enter a search query, and type characters to do further filtering of the results. Press RET to select a file to open. Use C-c f1 to see other available commands.

To configure notdeft-note-mode minor mode for use automatically in note buffers, add the required directory local variable to “~/.deft” by entering the command f6 a d l v, and by saving the resulting file.

For other ways to install, configure, and use NotDeft, see the rest of this page.

NotDeft's Deft Origins

NotDeft is derived from Deft version 0.3, but differs in several notable ways:

1. Rather than supporting a single, customizable deft-directory (tree) of note files, NotDeft supports a customizable notdeft-directories search path of directory trees.

   • If the search path includes multiple directories, then many file creation and management operations involve choosing a directory, making NotDeft not so deft.

2. NotDeft supports (optional) invocation of a notdeft-xapian-program, which uses the Xapian library to implement free-text search across note files, with convenient query syntax. The search is performed across all notdeft-directories, and further narrowing down of the result set can then be done incrementally by typing in a search string for purposes of filtering (as in Deft).

   • That is, when notdeft-xapian-program is set, the file browser of NotDeft lists Xapian search results instead of listing directory contents.
3. NotDeft includes multiple functions and commands intended to be usable from outside notdeft-mode, leading to more complex modalities of use than with Deft's list view centric operation.
4. NotDeft supports the existence of multiple notdeft-mode buffers within a single Emacs instance, so that one can view and operate on multiple search result sets of notes simultaneously.

NotDeft is not Deft—it aims for wide utility rather than the user experience of a Notational Velocity type application. The added complication of having two stages (query, then filter) to the process of looking for interesting files makes NotDeft less simple and intuitive than Deft, as does having multiple directories, modalities, and buffers.

NotDeft Installation

NotDeft can be installed either manually, or as a package.

Manual Installation from Source

First download the source code with the command

git clone https://github.com/hasu/notdeft.git

Go to the “notdeft” directory, and prepare the Emacs Lisp code for use with the command

make

Add the directory containing those files to the Emacs search path by adding

  (add-to-list 'load-path "/path/to/repo/of/notdeft")

to your Emacs startup file (e.g., “~/.emacs”). Also add

  (require 'notdeft-autoloads)

to the Emacs startup file, to make NotDeft available for on-demand loading.

With the above setup work done, NotDeft is available for launching from within Emacs with the command

M-x notdeft

While the above commands acquire, build, and set up NotDeft's Emacs Lisp code, they do not build and configure the C++-based Xapian backend; see Building the Xapian Backend and Configuring the Xapian Backend.

Installation as a Package with straight.el

The straight.el package manager is able to install NotDeft as a package directly from its source repository. If you have that manager correctly installed, then you can install NotDeft with the command

  (straight-use-package
   '(notdeft
     :type git :host github :repo "hasu/notdeft"
     :files ("*.el" "xapian")))

which should download NotDeft, generate its autoloads, and handle Emacs Lisp file byte-compilation.

While that command downloads and unpacks the Xapian backend source code, it does not build it or configure it; see Building the Xapian Backend and Configuring the Xapian Backend.

Installation from a Package File

Installing from Git is recommended where you wish to be sure that you are installing the most recent available version. Still, installation from a downloadable package file is also an option.

To install NotDeft as a package, first download the (chosen version's) package, and then install the downloaded TAR file with

M-x package-install-file

You can check whether the package has been installed by evaluating

(package-installed-p 'notdeft)

If so, information about the installation can be shown with

(describe-package 'notdeft)

No documentation is shown by that command, but it does show the location of the package's files, allowing navigation to the documentation.

One might also implement a command for opening something in the package. For example, the readme file can be opened with

  (defun notdeft-open-readme ()
    (interactive)
    (find-file
     (expand-file-name
      "README.org"
      (package-desc-dir
       (cadr (assq 'notdeft package-alist))))))

While installing the package does unpack the Xapian backend source code, it does not build it or configure it; see Building the Xapian Backend and Configuring the Xapian Backend.

Building the Xapian Backend

To enable Xapian search queries, you should build the notdeft-xapian C++ program in the “xapian” directory. On some systems simply going into that directory and typing

make

should do the trick, provided that the required tools and libraries have already been installed. Other systems may require more work not only on satisfying the dependencies, but also in finding the right C++ compiler incantation for building the program. (On some systems building it may not be feasible at all, and NotDeft's functionality will be more limited.)

Once a working compiler invocation command has been found, and the necessary C++ libraries have been installed, it is also possible to build the C++ program from within Emacs by using the included notdeft-xapian-make Emacs Lisp feature. To use it, set the variable notdeft-xapian-program-compile-command-format with the appropriate format string for the compilation command. In that format string the path of the executable comes first, and the path of the source file comes second. For example:

  (setq notdeft-xapian-program-compile-command-format "g++ -o %s %s -std=c++11 -Wall `pkg-config --cflags --libs tclap` `xapian-config --cxxflags --libs`")

With the feature appropriately configured you can then try issuing the command

M-x notdeft-xapian-compile-program

which should display any build errors if the executable cannot be built.

Building the Backend Automatically

The notdeft-xapian-make Emacs Lisp feature also includes a mechanism for building the Xapian backend program whenever it is out of date with respect to its sources, or does not exist at all. This can be particularly useful if you git pull a new version of “notdeft-xapian.cc”, and do not wish to worry about manually rebuilding the latest version.

For example, we might try to build notdeft-xapian on NotDeft startup as necessary:

  (add-hook 'notdeft-load-hook 'notdeft-xapian-make-program-when-uncurrent)

The notdeft-xapian-make-program-when-uncurrent function automatically sets the notdeft-xapian-program to the path of a successfully built program, so that it no longer needs to be specified otherwise.

NotDeft Configuration

Once the notdeft feature has been loaded, you can see and edit all of its configuration options and their documentation with

(customize-group "notdeft")

That command is also callable interactively as

M-x customize-group RET notdeft RET

The most essential settings are

notdeft-directories

to specify the location(s) of your notes

notdeft-xapian-program

to specify the path of the Xapian search tool

Specifying Note File Locations

In a simple case you would have a single directory (tree) of note file, specified by the notdeft-directories configuration variable, which you can configure with the command

M-x customize-variable RET notdeft-directories RET

For example:

(setq notdeft-directories '("~/all-my-notes"))

You can have multiple directories, which makes NotDeft use a bit harder, as you may at times get asked for a target directory for some file operations.

(setq notdeft-directories '("~/some-notes" "~/some-more"))

If your notes are not in a fixed directory, but you'd rather discover the directories programmatically, it may be convenient to set notdeft-directories in your startup file. For example:

(setq notdeft-directories (cons "~/notes" (file-expand-wildcards "~/*/notes")))

Sparse Directories

If you wish to include some additional text files into your searches, you may also explicitly specify files that reside outside any of the notdeft-directories. You must still specify a directory for a search index covering those files. In effect, you specify a sparse directory, since it is not scanned, but rather only explicitly specified files are considered to be NotDeft notes, if they exist.

To specify the index directories and any files within them, use the notdeft-sparse-directories configuration variable to specify directories and their file lists. For example:

  (setq notdeft-sparse-directories
	'(("~" .
	   ("projects/magnolisp/web/magnolisp-homepage.org"
	    "projects/notdeft/web/notdeft-homepage.org"))))

where all note file paths are specified relative to the search index containing directory, which should be a parent directory of all the specified notes.

The usual note manipulation operations (renaming, deleting, etc.) are not available for notes in sparse directories, which are not managed by NotDeft as such. The facility exists merely to support cases where you have important note files spread around project-specific directories, ones that you want to make accessible from within NotDeft. If you have a standard naming convention for such files, you can certainly resolve the list value programmatically:

  (setq notdeft-sparse-directories
	`(("~" . ,(mapcar
		   (lambda (file) (file-relative-name file "~"))
		   (file-expand-wildcards "~/projects/*/web/*-homepage.org")))))

Choosing the Note File Format

The default is to have the note filename notdeft-extension set to "org" to indicate the Org format. If you prefer some other note format, you should change that setting, which can be done with

M-x customize-variable RET notdeft-extension RET

The configured notdeft-extension is used by default when creating new notes, but a note collection can also use other extensions. There are none by default, but you can define such secondary extensions with

M-x customize-variable RET notdeft-secondary-extensions RET

For example, one might set these as

(setq notdeft-extension "txt")
(setq notdeft-secondary-extensions '("md" "org" "scrbl"))

Configuring the File Naming Convention

When creating a note file with the notdeft-new-file-named command, NotDeft automatically derives a name for the file based on the title that is provided for the note. The configuration option notdeft-notename-function determines how the name is derived.

The default setting is to use the notdeft-default-title-to-notename function to translate the title to a file basename. For example, the title “Rust (programming language)” translates into

rust-programming-language

The default implementation is suitable for titles with ASCII letters, and you probably want to pick a different implementation if your titles do not tend to use the English alphabet.

Where necessary, the configuration option notdeft-new-file-data-function provides even more control over the naming (and content) of new notes. Such a function could, for example, add some metadata to every new note's file name and/or content.

Configuring the Xapian Backend

To have NotDeft use the notdeft-xapian program you've built, you will have to specify its full path in the notdeft-xapian-program variable. You could use M-x customize-variable to set it, or simply

(setq notdeft-xapian-program "/path/to/notdeft-xapian")

Set the variable to the program's full absolute path, without any shorthands, as no shell expansion is performed on the path name—you may explicitly expand it using Emacs' expand-file-name function instead.

If you installed as a package, and built the notdeft-xapian executable in that location, then the appropriate setting may be

  (setq notdeft-xapian-program
	(expand-file-name
	 "xapian/notdeft-xapian"
	 (package-desc-dir
	  (cadr (assq 'notdeft package-alist)))))

Such code must appear after

(package-initialize)

See the other notdeft-xapian-* customization variables for configuring the Xapian indexing and searching behavior. Most notably:

• The configuration variable notdeft-xapian-max-results controls the maximum number of files to list in a notdeft-mode buffer. You may set it to 0 to always have all results displayed.
• The default is to order the results so that most recently edited files are listed first, but you may change this behavior by setting notdeft-xapian-order-by-time to nil, in which case Xapian's ranking mechanism is used instead.

NotDeft Mode

Running the notdeft command switches to a *NotDeft* buffer, creating one as necessary. Such a buffer's major mode is notdeft-mode. Buffers with that mode are read only, and cannot be edited directly, although most keys without modifiers do cause editing of the filter string.

Roughly, there are three kinds of things one can do in a *NotDeft* buffer:

1. set a query string to define a result set of notes
2. filter the result set by interactively editing a filter string
3. manipulate note files though NotDeft's commands

That is, finding an interesting set of notes is a two-step process: (1) enter a query to define a “topic area” of interest, using the Xapian query syntax; and then (2) narrow down that set interactively by typing in a list of substrings (in any order) that should match. It is possible to edit the query without modifying the filter string, and vice versa.

Querying and filtering in a *NotDeft* buffer.

The NotDeft Mode interface is optimized for editing the filter string. You can append characters to the filter by pressing regular symbol keys without modifiers. Other available commands include DEL, M-DEL, C-y, with familiar Emacs style behavior.

To enter a query, press TAB (or C-c C-o) to open a prompt for typing in the query. The query is then executed when you press RET.

To clear a query, you can

1. press TAB and enter the empty string, or
2. press S-TAB, or
3. C-u C-c C-c also works for clearing the query in addition to any filter string.

To manage the notes that are listed in the NotDeft Mode buffer, you can use mode-specific command, which are bound to the mode's C-c keymap. There are commands for renaming, deleting, and moving notes, for example. Press C-c f1 to see a full list of the commands bound to C-c.

To open a *NotDeft* buffer directly with a particular search query, use the command notdeft-open-query from any buffer.

Displaying Individual Filter String Matches

The filter string “emacs org mode” narrows a *NotDeft* buffer file list down only to the files that contain all of the substrings “emacs”, “org”, and “mode”. To see each of the matching positions within those files, consider entering the command C-c g (or M-x notdeft-grep-for-filter) to display the matching strings with highlighting. That command invokes the shell command grep (through the Emacs command grep), and displays the results in a separate buffer. This may fail to work if you system does not have a compatible grep executable on the search path.

Using Multiple NotDeft Mode Buffers

NotDeft allows multiple notdeft-mode buffers to exist at once, which may be useful if one wants to explore multiple sets of search results at once. Each NotDeft buffer has its own state, including a search query, filter string, default directory for creating new notes, etc.

Normally, executing the notdeft command only creates a new *NotDeft* buffer if one does not already exist—otherwise the command merely switches to an existing *NotDeft* buffer. It is possible to have the command always create a new notdeft-mode buffer by invoking it with a prefix argument, i.e., C-u M-x notdeft.

The notdeft-open-query command also accepts a prefix argument, to arrange for the search results to be listed in a new buffer. This behavior can also be made the default for that command by setting the configuration parameter notdeft-open-query-in-new-buffer to t. With that parameter set, the prefix argument's meaning is inverted, so that C-u M-x notdeft-open-query does not create an additional buffer.

The question of whether to create a new buffer does not apply to other search commands. Within a NotDeft buffer, the commands notdeft-query-edit and notdeft-query-clear merely replace the buffer's search result set, whereas the commands notdeft-lucky-find-file and notdeft-query-select-find-file do not use a NotDeft buffer for displaying their results.

For dealing with existing notdeft-mode buffers, there is a notdeft-switch-to-buffer command for interactively selecting a buffer and switching to it. It presents a choice list of buffer names in the minibuffer, and shows any query and filter strings associated with those buffers for better informed selection.

As for closing NotDeft buffers, the notdeft-quit command is bound to C-c C-q, and it can be invoked in three ways:

1. Without a prefix argument, it buries the current buffer.
2. With one prefix argument, it kills the current buffer.
3. With two prefix arguments, it kills all notdeft-mode buffers.

Four Emacs “windows” with different NotDeft buffers.

Displaying File Path Information

By default, NotDeft does not show any note directory or file names in its list view, but this behavior can be controlled by specifying a notdeft-file-display-function.

For example, we can display the name of each note's containing NotDeft (root) directory, with abbreviations for long directory names:

  (setq notdeft-file-display-function
	  (lambda (file w)
	    (when (> w 30)
	      (let* ((s (file-name-nondirectory
			 (directory-file-name
			  (notdeft-dir-of-file file))))
		     (s (pcase s
			  ("bibliography-notes" "bib")
			  ("homepage-notes" "hp")
			  (_ s)))
		     (s (if (> (string-width s) 12)
			    (truncate-string-to-width s 12)
			  s)))
		(concat " " s)))))

We refrain from displaying any directory information in cases where the Emacs window is very narrow (as indicated by the w argument), as otherwise there will be little space left for the note titles.

NotDeft mode with directory indicators.

NotDeft Note Mode

Invoking the notdeft command opens an Emacs buffer whose major mode is notdeft-mode. That mode displays a list of notes, and if you want the list to be automatically updated when a note file gets saved, you may want to enable the notdeft-note-mode minor mode for those files' buffers.

The sole purpose of notdeft-note-mode is to take care of keeping NotDeft's knowledge of the note collection up to date. Whenever a note file is saved, notdeft-note-mode sees to it that the search index is updated with the new file contents. NotDeft does not itself do anything to enable that mode, but rather the user should arrange for that to happen in some suitable way (see below for some suggestions). The benefit of this approach is that even if a note file then is open using a regular Emacs command (e.g., find-file), the editing buffer will notify NotDeft of any changes.

Enabling NotDeft Note Mode based on Major Mode

The simple approach is to always enable notdeft-note-mode for the major mode(s) that you use for editing notes. For example:

(add-hook 'org-mode-hook 'notdeft-note-mode)

This approach should be safe in that changes to files not residing in notdeft-directories get ignored by NotDeft. Still, the approach has the disadvantage that the minor mode indicator “¬D” does not tell you whether a note is actually a NotDeft note.

Enabling NotDeft Note Mode Locally to a Directory

Another solution is to try enabling notdeft-note-mode for every NotDeft directory in terms of per-directory local variables. For example, have your “.dir-locals.el” file state

((org-mode . ((mode . org)
              (mode . notdeft-note))))

This way of declaring both a major and minor mode appears to work at least in some versions of Emacs, although it may rely on undefined behavior.

Enabling NotDeft Note Mode based on a Directory-Local Variable

If enabling notdeft-note-mode directly in “.dir-locals.el” does not work or appeal to you, then it's possible to do the same thing indirectly, by using an actual per-directory local variable to indicate if the minor mode should be enabled. That is, you can have the “.dir-locals.el” file contain

((nil . ((notdeft-note-mode-auto-enable . t))))

The variable can be declared as

  (defcustom notdeft-note-mode-auto-enable nil
    "Whether to enable NotDeft note mode for a buffer."
    :type 'boolean
    :safe 'booleanp)
  (make-variable-buffer-local 'notdeft-note-mode-auto-enable)

To set that variable for a note directory, we can use the Emacs command

M-x add-dir-local-variable RET nil RET notdeft-note-mode-auto-enable RET t RET

Or, if we want to programmatically set the variable for all our notdeft-directories, we can use the code

  (dolist (dir notdeft-directories)
    (let ((default-directory dir))
      (add-dir-local-variable nil 'notdeft-note-mode-auto-enable t)))

Defining and setting the variable alone does not enable the mode, which we want to do only for specific file types, reflecting our notdeft-extension and notdeft-secondary-extensions configuration. If we only supported org-mode files, we would like to say something like

  (add-hook
   'org-mode-local-variables-hook
   (lambda ()
     (when notdeft-note-mode-auto-enable
       (notdeft-note-mode 1))))

We cannot just use org-mode-hook, as directory locals are not yet set at the time when the mode is enabled. What is needed is a later hook, which in the above is called org-mode-local-variables-hook.

We also have to get such hooks to run. Borrowing code from “phils” at Stack Overflow, we can get our org-mode-local-variables-hook run by defining and registering a new kind of hook as

  (defun run-local-variables-mode-hooks ()
    "Run hooks for `major-mode' with locals set.
  Like `run-mode-hooks', but run later, with any buffer and
  directory local variables set."
    (run-hooks (intern (concat (symbol-name major-mode)
			       "-local-variables-hook"))))
  (add-hook 'hack-local-variables-hook 'run-local-variables-mode-hooks)

The above solution gives us a “proper” way to enable the NotDeft note minor mode, and to do it only within directories that have a persistent NotDeft “signature” (in a “.dir-locals.el” file), and only for our chosen note-editing major modes.

Using NotDeft from Non-NotDeft Modes

Several of NotDeft's commands are autoloadable, and may be invoked from outside a *NotDeft* buffer. For example, to quickly find relevant notes when in another buffer, you might use

M-x notdeft-open-query

which then interactively asks for a search query for opening up in a NotDeft buffer. That command can of course be bound to a key.

A command similar to notdeft-open-query is

M-x notdeft-lucky-find-file

which also asks for a search query, but then proceeds to open up the most highly ranked result file directly, without going via a *NotDeft* buffer. This command is similar to find-file in Emacs, but avoids having to specify the path of the file you're interested in; instead, this approach to “file finding” relies on sufficiently unique titling or tagging of the notes involved.

NotDeft commands that are usable from outside notdeft-mode might be bound to key combinations for convenient access. To facilitate this, NotDeft provides a notdeft-global feature, which exports a keymap for such commands. That keymap can be bound to a prefix key. For example:

(require 'notdeft-global)
(global-set-key [f6] 'notdeft-global-map)

after which the command [f6] o should invoke the notdeft-open-query command in any mode that does not override the binding for F6 with something else.

Access from NotDeft Note Buffers

Some of NotDeft's commands have specific support for use from within NotDeft note buffers. For example, the notdeft-rename-file command can be useful for renaming a note file that was perhaps created without a proper name (e.g., by using C-c C-n). Having written a note in a current buffer, issue the command

M-x notdeft-rename-file

to enter a new basename for the file of that buffer. Any C-u prefix causes the default value to be derived from the title of the note, as extracted from the buffer contents. (The same command also works in a *NotDeft* buffer, affecting the currently selected file.)

Programmatic NotDeft Access

You might also implement additional commands in terms of the globally accessible commands and Emacs Lisp functions, for example for quickly listing documents tagged in a certain way:

  (defun my-open-todo-notes ()
    (interactive)
    (notdeft-open-query "tag:todo"))

An intended use case for NotDeft is to support other applications that wish to locate files in terms of search queries instead of path names. For example, suppose we are using an org-contacts command to look for contacts by name, and that command expects the org-contacts-files list to be set. In that scenario we might set that variable for it based on a suitable NotDeft search query:

  (setq org-contacts-files
	(notdeft-list-files-by-query
	 "!all ext:Org AND Email"))
  (org-contacts name)

Similarly, we might use org-agenda's org-todo-list command to list to-do entries, but resolving the org-agenda-files list on demand by looking for the “TODO” and “DONE” keywords in any Org files in our collection:

  (setq org-agenda-files
	(notdeft-list-files-by-query
	 "!all ext:Org AND (Todo OR Done)"))
  (org-todo-list)

NotDeft Note Syntax

NotDeft does not have much of a note syntax, although a subset of Org's syntax is supported in the form of in-buffer settings. The supported Org keywords are

• #+TITLE
• #+FILETAGS

A NotDeft-specific keyword is

• #+KEYWORDS

which is intended for tagging notes with keywords, in a way that does not set any tags for Org.

As for Org, the keyword names are case insensitive, so that one can write #+title instead of #+TITLE.

You can have in-buffer settings even if you do not use Org for your notes—the syntax for in-buffer settings is the same regardless of the markup language used in notes. Even in a plain “.txt” file, you can still specify #+KEYWORDS, for example.

Example Notes

No special markup is necessarily required:

  this is a title

  This is body text.

Comments can be included, and they are ignored when searching:

  # this is a comment
  this is a title

  This is body text.

Org mode's #+TITLE syntax is supported:

  # this is a comment
  #+TITLE: this is a title
  # this is a comment

  This is body text.

A note can be tagged, e.g., with the tags “some” and “tags”:

  #+TITLE: this is a title
  #+KEYWORDS: some tags

  This is body text.

Instead of the #+KEYWORDS syntax, we can use the Org standard #+FILETAGS syntax:

  #+FILETAGS: :some:tags:
  this is a title

  This is body text.

Stemming is used also on tags, and so the query “tag:tag” will find these two notes (assuming English stemming—see notdeft-xapian-language).

Whitespace is considered as a separator for tags, as are the delimiters “:”, “;”, and “,”. This means that the keyword declaration

  #+KEYWORDS: helsinki-vantaa places

is not matched by the search phrase “tag:"vantaa places"”. However, a hyphen still separates words, so that “tag:helsinki” and “tag:vantaa” and “tag:helsinki-vantaa” all match the first tag, which is semantically appropriate at least in this case.

Search Query Syntax

The usual Xapian search query syntax is available for NotDeft queries, with some additional query modifiers (see below). Operators such as AND, OR, and XOR are available, and they may also be written in lowercase (or mixed case) if notdeft-xapian-boolean-any-case is set to t. The NOT operator is also available if notdeft-xapian-pure-not is t. It is possible to query for a phrase by quoting the phrase (e.g., "Deft for Emacs"). To look for a search term without stemming, give it capitalized (e.g., "Abstract" will not match “abstraction”). Wildcards in search terms are not supported (trailing wildcards are supported by Xapian, but not enabled in NotDeft).

Prefixes

The following prefixes are supported by NotDeft:

file:

Indicates that the search term must appear in the (non-directory, non-extension) filename.

ext:

Indicates the string that must be the filename extension of the file (without the ".").

path:

Indicates that the search term must appear in the non-directory part of the file pathname, where the pathname is relative to the user's home directory.

title:

Indicates that the search term must appear in the title.

• Title is specified either as the first non-empty non-comment line, or as the file property (or Org mode “in-buffer setting”) #+TITLE. (Multiple #+TITLE lines are not supported.)

tag:

Indicates that the search term must appear among the tags given to the document.

• The tags for a note are specified either with the standard Org file property #+FILETAGS, or the custom file property #+KEYWORDS. (Org headline tags do not qualify.)

Query Modifiers

The following custom query syntax is supported:

!time

Prefix a query with !time to have the results sorted by decreasing file modification time, even if the notdeft-xapian-order-by-time configuration option is disabled.

!rank

Prefix a query with !rank to have the results sorted by decreasing relevance, regardless of the notdeft-xapian-order-by-time setting.

!file

Prefix a query with !file to have results sorted by (non-directory) file name, alphabetically, in decreasing order. Overrides all of the other sorting settings and modifiers.

!all

Prefix a query with !all to show all matching results. Note that unless you specify this modifier, the contents of a query result list may differ depending on how the results are sorted, since less highly ranked notes may get excluded.

A space character must be used to separate the above keywords from the rest of the query string.

The !file modifier might be useful for instance when you have file names such as “2017-01-01-0001.txt” and “2017-09-19-0123.txt”, and you would like to see them in chronological order by “creation time”, even if some of the files have been edited, and consequently have had their modification times changed.

Example Search Queries

It is simple to find all notes containing both the words Emacs and Org:

Emacs AND Org

If you have a lot of notes about Org mode, and few about other Emacs matters, it may be interesting to use

Emacs AND NOT Org

which works if the notdeft-xapian-pure-not option is set.

While you're often likely to be more interested in recent (or best maintained) notes, sorting by relevance can be useful particularly when there are multiple search terms: you may be more interested in seeing notes that contain all the terms instead of just one of them. You may use “!rank” to enable relevance-based ranking for a specific query:

!rank Emacs Org Deft

If, on the other hand, you use a single, common search term, and have a lot of documents, you may run into your notdeft-xapian-max-results limit, and miss out on some documents. In this case, you might use

!all Emacs

to list all documents mentioning Emacs.

If, unlike in the above case, you just want to see all documents that are about Emacs specifically, you may get more useful results with the query

title:Emacs

to only find documents whose title indicates that they concern Emacs. Or, to be more thorough, you might want to make sure you also find notes with the word Emacs in the filename:

title:Emacs OR file:Emacs

You can combine prefixes and “bracketed subexpressions”:

title:(Ayn AND Rand)

which will match both “Ayn Rand” and “Rand, Ayn” in a title.

Phrase searches are allowed for tags, and

tag:helsinki-vantaa
tag:"helsinki vantaa"
tag:(helsinki AND vantaa)

all match the tag “helsinki-vantaa”.

Filename extensions can be capitalized to avoid any stemming. For example, to find all “.org” documents that may contain open to-do entries, we might query with

!all ext:Org AND TODO

Command Popup Buffers

If it seems hard to remember the various NotDeft commands, one may wish to have a command selection dialog, similar to the one in Magit. For implementing such “helpful key bindings,” one can use Magit-Popup or Hydra, for instance. As an example, the “extras” directory of NotDeft's source repository contains a predefined hydra for NotDeft's mode-agnostic commands, provided by the notdeft-global-hydra feature. To bind [f6] to the hydra (instead of the notdeft-global-map keymap directly), one can use the configuration code

  (autoload 'notdeft-global-hydra/body "notdeft-global-hydra" nil t)
  (global-set-key [f6] 'notdeft-global-hydra/body)

There is also an optional hydra for notdeft-mode, which can be made available with code such as

  (autoload 'notdeft-mode-hydra/body "notdeft-mode-hydra")
  (eval-after-load "notdeft"
    '(define-key notdeft-mode-map (kbd "C-c h") 'notdeft-mode-hydra/body))

A NotDeft command “hydra” invoked from Org mode.

Org Mode Integration

NotDeft is somewhat specialized for managing notes in the Org format. If you do use Org mode for editing your notes, there are some Org-specific NotDeft commands available (for autoloading) in the notdeft-org feature.

Additionally, depending on your Org version, you may want to

(require 'notdeft-org8)

or

(require 'notdeft-org9)

in your Org startup code, to set up support for “deft:” and “notdeft:” links in org-mode. A “deft:” link names a note by its non-directory filename, whereas a “notdeft:” link contains a NotDeft Xapian search expression.

Org mode's org-store-link command may be used to capture any Xapian search in a NotDeft buffer, to be later inserted with org-insert-link. The notdeft-org feature also defines NotDeft-specific notdeft-org-link-existing-note and notdeft-org-link-new-file commands for inserting “deft:” links, either to an existing note or a new one.

The notdeft-org feature also defines a notdeft-org-store-deft-link command, which functions similarly to org-store-link, but stores a "deft:" link to the current note. In a NotDeft buffer, it stores a link to any selected note; and in a NotDeft note buffer, it stores a link to that buffer's note.

NotDeft allows a "deft:" link to also include a search option, which follows the filename, separated by ::. Search options are specified in the same way as for "file:" links. For example:

[[deft:notdeft-homepage.org::*Note Archival]]
[[deft:notdeft-homepage.org::#capture-protocol]]

NotDeft has some optional support for the Org property syntax, which can be enabled by setting the variable notdeft-allow-org-property-drawers to a non-nil value. Enabling that option makes it so that any top-level PROPERTIES drawer appearing at the beginning of a note is treated as a comment. The title of the note

  :PROPERTIES:
  :CUSTOM_ID: my-custom-id
  :END:
  Note title
  Note body.

can be either ":PROPERTIES:" or "Note title", depending on whether NotDeft is configured to recognize PROPERTIES drawers.

Using NotDeft and Org Mode as a Desktop Wiki Engine

It is “deft:” links in particular that allow NotDeft to be used as a desktop wiki, linking documents by topic, where a topic is named by the non-directory name of a note file. For “deft:” links to consistently resolve to the same note, you should name your note files uniquely.

For example, when following the link

[[deft:notdeft.org]]

NotDeft will look for a “notdeft.org” file anywhere in the note collection, and open the first match.

A benefit of that “deft:” link semantics is that using the command

M-x notdeft-move-file

to move a note file into a different directory does not cause any “deft:” link to break, whereas regular “file:” links may break.

To conveniently create a dedicated note for a given topic in an Org-mode buffer, and also link to that note at the same time, highlight the title (and link description) of that topic so that it becomes the active region, and then issue the command

M-x notdeft-link-new-file

For example, if you've highlighted the text “desktop wikis”, the command will offer to create a note of the same title, derive a filename for it based on the title, and replace the region with a “deft:” link to it. (The command is defined by the notdeft-org feature.)

Quick Note Capture

To quickly create a new note file from any buffer, you can use

M-x notdeft-new-file

That command is also bound to C-n in notdeft-global-map, and if that keymap is bound to the prefix [f6], for example, then you can create a new note with the key combination [f6] C-n.

Org mode has its own “capture” mechanism, and you can certainly configure capturing into a file that resides in a NotDeft directory. For example:

(setq org-directory "~/notes") ;; default Org files location
(setq notdeft-directories (list org-directory)) ;; NotDeft search path
(setq org-default-notes-file (concat org-directory "/notes.org"))
(global-set-key [f7] 'org-capture)

which defines "~/notes" as the sole NotDeft directory, and has the key F7 initiate an org-capture, by default into the file "~/notes/notes.org". After completing capture, you can go back to the previously captured item with

C-u C-u M-x org-capture

The capture facility supports the definition and use of org-capture-templates for different purposes.

A caveat with Org capturing is that unless you have already opened the capture file under NotDeft, any newly captured items may not immediately get noticed by NotDeft. To ensure that NotDeft is aware of any changes, one might arrange for the capture file to include file variables for enabling the notdeft-note-mode minor mode for any buffers opened for that file. Setting directory local variables are another option.

A more involved option is to write custom commands which enable the minor mode for the capture file, for example with

(notdeft-register-file org-default-notes-file)

Note that different org-capture-templates may define different capture locations. Consequently, it may be appropriate for the templates themselves to embed code for performing the registration (e.g., as shown in the capture from Firefox section).

Adding Attachments to Notes

NotDeft has a simple mechanism to support “attaching” files to notes, one that is agnostic to the note file format. If you have a note file

~/notes/deft-for-emacs.txt

you can use the command C-c S to move the file into a subdirectory of the same name, so that the file's pathname becomes

~/notes/deft-for-emacs/deft-for-emacs.txt

Now you can copy/move/link any attachments for the note into that subdirectory, and it is convenient to move the note together with its attachments using a regular file manager.

To move a note from within *NotDeft*, the command C-u C-c m can be used to move it under another NotDeft root directory, where the prefix C-u assures NotDeft that the file really is to be moved together with its subdirectory.

When the attachments reside in the same directory as the note itself, in Org mode it is then easy to add a “file:” link to any attachment with the command C-u C-c C-l. For example, if the attachment directory contains a file named “2017-01-01-0001.JPG”, then a “file:” link to it would be simply

[[file:2017-01-01-0001.JPG]]

and the command C-c C-x C-v can be used to toggle inline display of images.

Org itself has its own attachment management mechanism, whose action menu is bound to C-c C-a. This mechanism allows an attachment directory to be associated with an Org heading (as identified by information stored within the heading's properties), and thus the NotDeft note file itself can reside directly within a NotDeft root directory. Org has no command for moving an Org file together with its attachments, however.

To make the Org mechanism compatible with the NotDeft mechanism, one can store the attachments in the same (sub)directory as the note file itself, by specifying that directory with the ATTACH_DIR property. For example:

* Bergen, Norway                              :ATTACH:
:PROPERTIES:
:ATTACH_DIR: ./
:Attachments: 2017-01-01-0001.JPG 2017-09-19-0123.JPG
:END:

This way it is still convenient to move a note together with its attachments, and Org commands such as C-c C-a o (for opening the attachments) can still be used.

Note Archival

To archive away a note so that its contents will no longer be included in a search, one can press C-c C-a from within *NotDeft*. This is a note format agnostic archival method, as the entire note file gets moved into a notdeft-archive-directory, with the default name of

"_archive"

meaning that a note file whose original path is

~/notes/deft-for-emacs.txt

would get moved to

~/notes/_archive/deft-for-emacs.txt

Any directories whose names begin with an underscore will be excluded from Xapian searches, and thus such an archived note will no longer clutter search results.

In Org mode one can use Org's own archival mechanism to archive just a part of a note document subtree, and the archival file will also be excluded from Xapian searches, provided that its filename extension is not notdeft-extension or one of the notdeft-secondary-extensions. Org's default extension is

org_archive

which by default is not an extension recognized by NotDeft.

Capturing Data from External Applications

The org-protocol feature of Org mode provides a way for external applications to interface with Emacs and Org, and that mechanism can also be adopted for capturing data into NotDeft. For example, data can be sent from Firefox to NotDeft using the predefined store-link and capture protocols.

The mechanism works by the external application invoking emacsclient, and for this to work you should have an Emacs server running in the Emacs instance you want to use to receive data into NotDeft. A server can be started by evaluating

(server-start)

org-protocol Content Type in Firefox

To configure Firefox to support the org-protocol: scheme, first open about:config, and add a boolean property

network.protocol-handler.expose.org-protocol false

Then craft an HTML file such as

  <html>
    <body>
      <a href="org-protocol://store-link?url=URL&title=TITLE">link</a>
    </body>
  </html>

and open that file in Firefox, and click the link, after which a “Launch Application” dialog is presented. “Choose other Application”, tick the box “Remember my choice for org-protocol links”, and specify emacsclient as the executable.

That application selection can later be modified from Firefox “Preferences” / “Applications”. If required, the “Content Type” should be removable at least by editing the “mimeTypes.rdf” file in the Firefox profile.

store-link from Firefox

There is nothing NotDeft specific about the store-link Org protocol, as it merely stores a link to the Emacs kill-ring for yanking. To configure Firefox to support the protocol, just add a suitable bookmarklet (e.g., to the “Bookmarks Toolbar”). The bookmark “Location” can be specified as

javascript:location.href='org-protocol://store-link?url='+encodeURIComponent(document.location)+'&title='+encodeURIComponent(document.title);void(0);

By selecting that bookmark a link to the current page can be sent to Emacs. Its URL can then be inserted in Emacs with C-y. A full Org link in turn can be inserted with

M-x org-insert-link

which is bound to C-c C-l in Org.

capture from Firefox

The capture protocol, in turn, allows for web page content and metadata to be captured from Firefox into Emacs. Configuring the capture protocol for use with NotDeft is slightly more involved than in the case of store-link, as we must choose what page data to store, and where in our NotDeft note collection to store it.

Suppose we wish to store any currently selected text, along with the URL of the containing page, and a capture timestamp. Suppose also that we wish to store it into a file whose name is derived from the page title, so that if we capture multiple times from the same page, then all of the captured text snippets will end up in the same note file.

In that case the Firefox bookmarklet for sending over the required information can for example be

javascript:location.href='org-protocol://capture?template=w&url='+encodeURIComponent(document.location)+'&title='+encodeURIComponent(document.title)+'&body='+encodeURIComponent(window.getSelection());void(0);

where we have given the name “w” for the Org capture template.

We must also define that template as one of our org-capture-templates, and the definition can be

  (require 'org-protocol)

  (setq org-capture-templates
	'(("w" "capture selection into NotDeft" plain
	   (file (lambda ()
		   (notdeft-switch-to-file-named
		     (plist-get org-store-link-plist :description))))
	   "%l\non %u\n\n%i"
	   :empty-lines-before 1)))

This definition assumes that the link :description is available from org-store-link-plist, and that it corresponds to the document.title; this may be undocumented functionality, but works in Org mode 9.1.1. The notdeft-switch-to-file-named derives a filename from the description, creates that file if it doesn't yet exist, and returns the complete file name.

Troubleshooting

When Search Queries Are Not Yielding Expected Results

Try doing the following in a *NotDeft* buffer:

1. Press TAB (or M-x notdeft-query-edit) to be prompted for a Xapian query.
2. If nothing happens when you press TAB, then you have probably not configured a value for notdeft-xapian-program. Assign a value to that variable.
3. Having pressed TAB, enter a query string at the prompt, one that should match some notes, and press RET.
4. If that reports "Found no notes", or an unexpected set of notes, then your search index may not be up-to-date, perhaps due to filesystem changes outside of NotDeft. Invoke the command M-x notdeft-refresh (i.e., C-c C-x g) to refresh the search index.
5. If you suspect that your search index may be corrupt or incompatible in some way, you may invoke the command M-x notdeft-reindex (i.e., C-c C-x r) to fully rebuild the search index, instead of just refreshing it.

6. If you see unexpected behavior after setting a search query, ensure that the notdeft-xapian-program variable names the complete and fully expanded path of a working executable. It may be worth trying to run the program directly, and seeing what it says. For example:

   /path/to/notdeft-xapian search -q 'Emacs OR Vi' ~/.deft
   

7. If your search query includes prefix terms such as “title:Emacs”, and you do not get all the expected matches, then make sure that any lines before any #+TITLE (or, #+KEYWORDS, etc.) are either whitespace only or begin with “#”. While the Org markup language allows in-buffer settings to appear anywhere in a file, NotDeft only scans the beginning of each file for such settings.
8. If all else fails, a tool such as xapian-delve may be used to inspect the contents of the search index to see which terms it actually contains.

See Also

{{< taggedpagelistexceptself "notdeft" >}}