vikix.GuideInstallSource

12 Nyxt, the browser you change in Lisp

Nyxt is a web browser written in Common Lisp, the language StumpWM is written in. Its keys, its commands and how it loads pages are all Lisp, and you can change them while it runs, the way you can change the desktop. Firefox stays your browser for the web at large; Nyxt is the one Vikix uses for its guides and docs, and the one to shape to your own reading.

It comes with the feature lisp-apps, beside Lem and McCLIM’s Listener:

vikix add lisp-apps          # Nyxt, Lem and the Listener
vikix lisp-apps status       # what's here, and its colours
vikix remove lisp-apps       # takes them away again; your config stays

Nyxt is in the launcher (Super+d). Everything below is safe to try: your config is a file of your own, and the snapshot history keeps it.


12.1 What Vikix sets up in Nyxt

Nyxt’s own commands are one key away: Ctrl+Space asks for a command by name (type a few letters of it), and Ctrl+Space, then manual, opens Nyxt’s manual.


12.2 Guides and docs in Nyxt

The menu entries and the docs alias run vikix-docs-open, which hands local pages to Nyxt as file:// addresses. If Nyxt is already open, the page joins it as a new tab, and StumpWM brings its window forward from whatever workspace it’s on. Without Nyxt, vikix-docs-open uses xdg-open, your default browser.

To send guides and docs to another browser, name it in ~/.config/vikix/docs-browser, one word on a line:

echo firefox > ~/.config/vikix/docs-browser

Delete the file to have Nyxt again. The rest of the web is unaffected either way: links from other programs go to your default browser (~/.config/mimeapps.list).

Every document, as a page. Super+F2 searches every document on the machine (these guides, man pages, your projects’ documents and PDFs, your notes, the API references in Zeal’s docsets, and links to the web, which say so) and shows the hits in a list. With Nyxt the same search is also a page: Super+m → Every document as a page in Nyxt, vikix docs page WORDS in a terminal, or Ctrl+Space, then vikix-docs, in Nyxt itself. The hits are grouped by whose they are, each with an Open button and, where there is another place to read it, a second one named for it (In Emacs, In a terminal); Find… searches again. Nyxt is started if it isn’t running, and its window comes to the front.


12.3 Nyxt’s colours

Nyxt reads the desktop’s palette, ~/.config/vikix/theme/palette (the key=#rrggbb lines vikix theme writes for every program), each time it starts, and builds a Nyxt theme from it. vikix theme NAME also asks a running Nyxt to read it again: the status bars and the message line change at once, and new tabs and prompts follow.

Pages keep their own colours. If Nyxt’s dark mode is on (its settings can switch it on for every page), it darkens pages too; No dark mode on the guide pages keeps it off Vikix’s guide.

To keep Nyxt’s own colours, take the line that loads Vikix’s part out of your config.lisp. vikix lisp-apps status says which you have:

nyxt's colours follow vikix theme (~/.config/nyxt/config.lisp loads Vikix's part)

12.4 The home page

Nyxt starts on a page of Vikix’s, and Ctrl+t opens a new tab on it: the logo, buttons to the Nyxt guide, the docs catalogue, Nyxt’s manual, the list of every key and its settings, and the keys that matter most in Nyxt’s default scheme (the Firefox-like one Vikix leaves on), each with the name of its command, since Ctrl+Space takes names. Ctrl+Space, then vikix-home, brings it back from anywhere. The logo is the site’s, light or dark for the theme’s background, read from ~/vikix.

To have Nyxt’s own page instead, in config.lisp after the line that loads Vikix’s part:

(define-configuration browser
  ((default-new-buffer-url (quri:uri "nyxt:new"))))

Any address does there: (quri:uri "https://example.org").


12.5 Your Nyxt config

~/.config/nyxt/config.lisp

Whose: yours
What: Your settings, keys and commands, in Lisp (package nyxt-user). Copied once from Vikix’s starter

~/.local/share/vikix/nyxt/vikix.lisp

Whose: Vikix’s
What: The colours, the remote calls vikix theme makes, Swank, the docs catalogue’s page and the home page. A link into ~/vikix, kept current by vikix update: don’t edit it

The starter config.lisp is a few lines: the in-package line, the load of Vikix’s part, and room for yours below. A line of yours placed after the load wins over Vikix’s. Two you may want:

;; No Swank in Nyxt (see "Nyxt from Emacs").
(setf *vikix-swank-port* nil)

;; A light theme's pages unchanged: no dark mode on any page.
(define-configuration web-buffer
  ((default-modes (remove 'nyxt/mode/style:dark-mode %slot-value%))))

Nyxt reads config.lisp and Vikix’s part when it starts, so a change takes effect at the next start (or at once, evaluated from Emacs). The snapshot history keeps all of ~/.config/nyxt: vikix changes shows what changed there, and vikix undo takes it back. Nyxt writes settings of its own beside yours, in auto-config.3.lisp (what you change from its menus); the history keeps that too.


12.6 Nyxt from Emacs

Swank is the program inside a running Lisp that lets Emacs’s SLIME talk to it: evaluate code, look at values, jump to the source. StumpWM has one on port 4004 (see How it fits together); Vikix’s part starts one in Nyxt on 4006.


12.6.1 Connecting to Nyxt

  1. In Emacs, M-x slime-connect RET 127.0.0.1 RET 4006. SLIME sends the password by itself. If it says the SLIME and Swank versions differ, answer y: it works.
  2. You’re at a REPL inside the browser, in the package nyxt-user. Try (render-url (url (current-buffer))), the address of the page you’re on.

It answers only on your own machine, and only with the password in ~/.slime-secret, the same file StumpWM’s Swank uses (Vikix makes it). Without that file, Nyxt’s Swank doesn’t start at all. A client that sends a wrong password, or none within 5 seconds, is turned away without taking Swank down for the next one.

Emacs can be connected to StumpWM and Nyxt at once. A Lisp buffer sends to the default connection, the last one made. M-x slime-list-connections shows them: the Port column tells them apart (4006 Nyxt, 4004 StumpWM) and * marks the default. On a line there, d makes it the default and C-k closes it. M-x slime-cycle-connections switches too.


12.6.2 Evaluating in a Lisp buffer

Keep your experiments in a Lisp file (say ~/nyxt.lisp) that starts with:

(in-package #:nyxt-user)

SLIME reads that line to know the package. Without it the forms run in cl-user, and Nyxt’s names, such as define-command-global, aren’t found.

C-M-x

The top-level form around the cursor: the usual one for a command

C-c C-c

The same, compiled, with warnings shown on the code

C-x C-e

The expression before the cursor, such as (title (current-buffer)); the result shows in the echo area

C-c C-r

The region

C-c C-k

The whole file

~/.config/nyxt/config.lisp starts with the same line, so you can work on it the same way. C-c C-k on it is safe: Vikix’s part starts Swank only once.

A few things to try with C-x C-e:

  • (mapcar #'title (buffer-list)): the titles of your open pages.
  • (render-url (url (current-buffer))): the current address.
  • M-. on any Nyxt name (render-url, print-status) jumps into Nyxt’s own source, in /usr/share/nyxt/source/; M-, comes back.

12.6.3 Without Emacs

nyxt --remote --quit --eval '(...)' runs Lisp in the running Nyxt, over its socket in your runtime folder ($XDG_RUNTIME_DIR/nyxt/nyxt.socket, yours alone). It’s what vikix theme uses. Whatever the code prints goes to Nyxt’s log, ~/.local/share/nyxt/nyxt.log, not to your terminal.


12.7 Examples to try

Each of these was run against Nyxt 3.11.8, Void’s package. Paste one into your Lisp buffer and C-M-x it to try it now; put it in config.lisp, below the load line, to keep it.


12.7.1 Open the page in Firefox

Some sites work badly in WebKit, the engine Nyxt draws pages with. This command hands the page to Firefox:

(define-command-global open-in-firefox ()
  "Open the current page in Firefox."
  (uiop:launch-program (list "firefox" (render-url (url (current-buffer)))))
  (echo "Sent to Firefox: ~a" (title (current-buffer))))

It’s a command as soon as it’s evaluated: Ctrl+Space, open-in-f, Enter. Change it and C-M-x again to replace it, with no restart.


12.7.2 No dark mode on the guide pages

An auto-rule changes a page’s modes when its address matches. This one keeps dark mode off Vikix’s guide, which has colours of its own, and off the docs in ~/dev, while other pages stay dark:

;; Vikix's guide and ~/dev's docs have their own colours: no dark mode there.
(define-auto-rule
    `(match-regex ,(format nil "^file://~a\\.local/share/vikix/guide/"
                           (namestring (user-homedir-pathname)))
                  ,(format nil "^file://~adev/" (namestring (user-homedir-pathname))))
  :excluded '(nyxt/mode/style:dark-mode))
  • The first part is the test. It’s evaluated, and what comes back is called with the page’s address. match-regex takes one or more patterns; the others are match-domain, match-host, match-port, match-scheme and match-url.
  • :excluded turns modes off on matching pages, :included turns them on.
  • Rules apply as a page loads: reload it (Ctrl+r, or F5) to see one.

12.7.3 Page tools

A file of commands for what’s on a page, made on a Vikix laptop as an example of what a few dozen lines of Lisp can add. Save the code below as ~/.config/nyxt/page-tools.lisp and load it from config.lisp, below Vikix’s line:

(load (merge-pathnames "page-tools.lisp" (uiop:xdg-config-home "nyxt/")))

Then each is in Ctrl+Space:

save-page-images

What you get: Every picture on the page, in the page’s order, each once
Where it goes: ~/Pictures/Nyxt/<page title>/

save-page-tables

What you get: Every table on the page as CSV, one file each, for a spreadsheet
Where it goes: ~/Documents/Nyxt/<page title>/table-N.csv

save-page-code

What you get: Every code block on the page as a file, the extension from the language the page names
Where it goes: ~/Documents/Nyxt/<page title>/code-N.py (.sh, .lisp, …, .txt)

copy-page-code

What you get: One code block, picked from a list (its first line, language and length), copied
Where it goes: The clipboard

jump-to-heading

What you get: The page’s headings as a list; pick one and the page scrolls there
Where it goes: The page

reader-view

What you get: The page’s article alone, in Nyxt’s own style, links still working
Where it goes: A page of its own in Nyxt

page-to-markdown

What you get: The page’s article (no menus or footers, when the page marks its article) as Markdown, links still working
Where it goes: ~/Documents/Nyxt/<page title>.md, and the clipboard

page-to-epub

What you get: The page as an e-book, pictures inside, for an e-reader
Where it goes: ~/Documents/Nyxt/<page title>.epub

tabs-to-epub

What you get: Every open tab as one e-book, a chapter each: a reading list
Where it goes: ~/Documents/Nyxt/Reading list <date>.epub

clip-selection

What you get: The selected text as a quote, with the date, the page’s title and its link
Where it goes: The end of ~/Documents/Nyxt/clips.md, and the clipboard

clip-to-project-log

What you get: The selected text as a quote, with the page’s title and link, as today’s entry in a project’s log (the project picked from your list); nothing selected: the title and link alone
Where it goes: The project’s log.md, through vikix project log

add-page-to-notes

What you get: The page as Markdown among your notes, and indexed, so note ask and note find see it
Where it goes: Web/<page title>.md in the folder note index reads

summarize-page

What you get: A short summary in bullet points, and who the page is for
Where it goes: A page of its own in Nyxt, kept until Nyxt quits

show-url-qrcode

What you get: The page’s address as a QR code, to open it on a phone
Where it goes: A page of its own (Nyxt’s own command: nothing to add)

Things to know about them:

  • They need pandoc for Markdown, e-books, the reader view and the quotes (vikix add cli-extras), llm for the summary (vikix add llm), vikix notes for the page into your notes (vikix add notes, then note index ~/Notes once: the page goes into that folder’s Web/, and note index runs again by itself, in the background), and a project for the log (vikix project new NAME).
  • The tables come as the page shows them: a cell’s text on one line, quotes doubled, a field in quotes when it holds a comma; a table inside a table is read as part of the outer one.
  • A code block’s language comes from a class like language-python on its <pre> or <code>, as most sites write it; a block with no language is .txt.
  • The summary goes to Claude. summarize-page sends the page’s text to Anthropic through llm, with the key from vikix ai key set anthropic, and takes seconds. To keep pages on the machine, set *summary-model* to nil in config.lisp, after the page tools’ line: llm’s default model then answers (a local one, if that’s what you chose), free and private, but it can take minutes on a long page and says less. For a small local model, set *summary-max-chars* near 6000 too.
  • Pictures behind a login don’t come. save-page-images fetches pictures itself, without the browser’s cookies.
  • Pages that mark no article (<article> or <main>) come whole, menus and all.
  • Where things go can be changed: *page-tools-folder* (Markdown, e-books, clips) and *clips-file*, set in config.lisp after the page tools’ line.

For the curious, how they read the page: (document-model buffer) is Nyxt’s parsed copy of it (made with plump); clss:select finds elements in it with CSS selectors; plump:clone-node copies a part, so the page itself isn’t changed; make-addresses-whole, in the file, makes links and pictures absolute so they still work away from the page; and ps-eval runs JavaScript written in Lisp (Parenscript) in the page, here to read the selection.

The file:

(in-package #:nyxt-user)

;;; Page tools: a page's images, tables (as CSV) and code blocks (as files,
;;; or one copied), the page as Markdown or an e-book (or all open pages as
;;; one), a reader view, a jump to a heading, the selection clipped to a notes
;;; file or a project's log, the page into your notes' index, and a summary
;;; by llm. The QR code of a page is Nyxt's own: show-url-qrcode.

(defvar *page-tools-folder* (merge-pathnames "Documents/Nyxt/" (user-homedir-pathname))
  "Where Markdown, e-books and clips go.")

(defvar *clips-file* (merge-pathnames "clips.md" *page-tools-folder*)
  "The notes file clip-selection adds to.")

(defvar *summary-model* "claude-sonnet-5"
  "The llm model summarize-page asks: Claude, through llm-anthropic and your
key (vikix ai key set anthropic). \"llama3.2:3b\" for the local one (slow on
the CPU: minutes a page, and less sharp); nil for llm's default.")

(defvar *summary-max-chars* 50000
  "How much of a page summarize-page sends: about 12,000 tokens, which
Claude reads in seconds. For a local model, 6000: Ollama's default context
cut a 4,000-token page short, and the CPU took five minutes to read it.")

;;; --- Helpers ----------------------------------------------------------------

(defun page-file-name (title)
  "TITLE as a file name: no slashes, no leading dot, not too long."
  (let ((name (string-trim " ." (substitute-if #\- (lambda (c) (find c "/\\:*?\"<>|")) title))))
    (subseq name 0 (min 80 (length name)))))

(defun page-url (buffer address)
  "ADDRESS (as written in the page, maybe relative) made whole against BUFFER's URL."
  (quri:render-uri (quri:merge-uris (quri:uri address) (url buffer))))

(defun image-address (img)
  "IMG's picture, as written in the page. Lazy-loading pages keep it in
data-src, with a placeholder (a data: URI) in src."
  (find-if (lambda (s) (and s (plusp (length s)) (not (str:starts-with-p "data:" s))))
           (list (plump:attribute img "src") (plump:attribute img "data-src"))))

(defun local-path (address)
  "The file behind a file:// ADDRESS, or ADDRESS as it is."
  (let ((uri (quri:uri address)))
    (if (string= "file" (quri:uri-scheme uri))
        (quri:url-decode (quri:uri-path uri))
        address)))

(defun pandoc (input &rest args)
  "Run pandoc on the string INPUT with ARGS; its output as a string."
  (with-input-from-string (in input)
    (uiop:run-program (cons "pandoc" args) :input in :output :string)))

(defun make-addresses-whole (buffer node &key local-images)
  "In NODE (a part of BUFFER's page, copied), links and pictures made whole,
so they still work away from the page. LOCAL-IMAGES: file:// pictures as
plain paths, which pandoc reads when it makes an e-book."
  (loop for a across (clss:select "a[href]" node)
        do (ignore-errors
            (setf (plump:attribute a "href") (page-url buffer (plump:attribute a "href")))))
  (loop for img across (clss:select "img" node)
        for src = (image-address img)
        when src do (ignore-errors
                     (let ((whole (page-url buffer src)))
                       (setf (plump:attribute img "src")
                             (if local-images (local-path whole) whole)))))
  node)

(defun page-html (buffer &key local-images)
  "BUFFER's article (else its main part, else all of it) as HTML, with
make-addresses-whole. A copy: the page isn't changed."
  (let* ((dom (document-model buffer))
         (part (or (loop for selector in '("article" "main" "body")
                         for found = (clss:select selector dom)
                         when (plusp (length found)) return (elt found 0))
                   dom))
         (node (plump:clone-node part t)))
    (plump:serialize (make-addresses-whole buffer node :local-images local-images) nil)))

(defun page-markdown (buffer &key local-images (shift 0))
  "BUFFER's article as Markdown. SHIFT moves its headings down that many levels."
  (pandoc (page-html buffer :local-images local-images)
          "-f" "html" "-t" "gfm-raw_html" "--wrap=none"
          (format nil "--shift-heading-level-by=~d" shift)))

(defun today ()
  (multiple-value-bind (s m h day month year) (get-decoded-time)
    (declare (ignore s))
    (format nil "~d-~2,'0d-~2,'0d ~2,'0d:~2,'0d" year month day h m)))

;;; --- Images -----------------------------------------------------------------

(define-command-global save-page-images ()
  "Save every image on the page into ~/Pictures/Nyxt/<page title>/."
  (let* ((buffer (current-buffer))
         (folder (merge-pathnames (format nil "Pictures/Nyxt/~a/" (page-file-name (title buffer)))
                                  (user-homedir-pathname)))
         (addresses
           (remove-duplicates
            (loop for img across (clss:select "img" (document-model buffer))
                  for src = (image-address img)
                  when src collect (page-url buffer src))
            :test #'string= :from-end t))
         (saved 0))
    (ensure-directories-exist folder)
    (loop for address in addresses
          for n from 1
          for name = (or (car (last (str:split "/" (quri:uri-path (quri:uri address)))))
                         "image")
          do (handler-case
                 ;; Numbered, so they keep the page's order, and two
                 ;; pictures with the same name don't overwrite each other.
                 (let ((to (merge-pathnames (format nil "~3,'0d-~a" n name) folder)))
                   (if (string= "file" (quri:uri-scheme (quri:uri address)))
                       (uiop:copy-file (local-path address) to)   ; a local page
                       (alexandria:write-byte-vector-into-file
                        (dex:get address :force-binary t) to :if-exists :supersede))
                   (incf saved))
               (error (e) (log:warn "Couldn't save ~a: ~a" address e))))
    (echo "Saved ~d of ~d images to ~a" saved (length addresses) (namestring folder))))

;;; --- Markdown and e-books ---------------------------------------------------

(define-command-global page-to-markdown ()
  "Save the page's article (or the whole page) as Markdown, in
~/Documents/Nyxt/<page title>.md, and copy it to the clipboard."
  (let* ((buffer (current-buffer))
         (file (merge-pathnames (format nil "~a.md" (page-file-name (title buffer)))
                                *page-tools-folder*))
         (markdown (format nil "# ~a~%~%Source: <~a>~%~%~a"
                           (title buffer) (render-url (url buffer)) (page-markdown buffer))))
    (ensure-directories-exist file)
    (alexandria:write-string-into-file markdown file :if-exists :supersede)
    (copy-to-clipboard markdown)
    (echo "Markdown saved to ~a (and copied)" (namestring file))))

(defun write-epub (title chapters file)
  "An EPUB at FILE called TITLE, from CHAPTERS: a list of (title url buffer)."
  (let ((markdown
          (with-output-to-string (out)
            (dolist (chapter chapters)
              (destructuring-bind (name address buffer) chapter
                ;; Each page a chapter: its title the top heading, its own
                ;; headings one level down.
                (format out "# ~a~%~%Source: <~a>~%~%~a~%~%"
                        name address (page-markdown buffer :local-images t :shift 1)))))))
    (ensure-directories-exist file)
    ;; Pandoc fetches the web pictures itself and puts them in the book.
    (pandoc markdown "-f" "gfm" "-t" "epub3" "-o" (namestring file)
            "--metadata" (format nil "title=~a" title)
            "--metadata" "lang=en")
    file))

(define-command-global page-to-epub ()
  "Save the page as an e-book, ~/Documents/Nyxt/<page title>.epub, pictures
included."
  (let* ((buffer (current-buffer))
         (file (merge-pathnames (format nil "~a.epub" (page-file-name (title buffer)))
                                *page-tools-folder*)))
    (echo "Making an e-book of ~a..." (title buffer))
    (write-epub (title buffer) (list (list (title buffer) (render-url (url buffer)) buffer)) file)
    (echo "E-book saved to ~a" (namestring file))))

(define-command-global tabs-to-epub ()
  "Save every open web page as one e-book, a chapter each:
~/Documents/Nyxt/Reading list <date>.epub."
  (let* ((buffers (remove-if-not (lambda (b) (and (typep b 'web-buffer)
                                                  (member (quri:uri-scheme (url b))
                                                          '("http" "https" "file")
                                                          :test #'equal)))
                                 (buffer-list)))
         (title (format nil "Reading list ~a" (subseq (today) 0 10)))
         (file (merge-pathnames (format nil "~a.epub" title) *page-tools-folder*)))
    (if (null buffers)
        (echo "No web pages open")
        (progn
          (echo "Making an e-book of ~d pages..." (length buffers))
          (write-epub title
                      (mapcar (lambda (b) (list (title b) (render-url (url b)) b)) buffers)
                      file)
          (echo "E-book of ~d pages saved to ~a" (length buffers) (namestring file))))))

;;; --- Clipping the selection -------------------------------------------------

(define-command-global clip-selection ()
  "Add the selected text, as a Markdown quote with the page's title and
address, to ~/Documents/Nyxt/clips.md, and copy it."
  (let* ((buffer (current-buffer))
         (quote (selection-quote buffer)))
    (if (null quote)
        (echo "Select some text first")
        (let ((clip (format nil "## ~a, [~a](~a)~%~%~a~%~%"
                            (today) (title buffer) (render-url (url buffer)) quote)))
          (ensure-directories-exist *clips-file*)
          (with-open-file (out *clips-file* :direction :output
                                            :if-exists :append :if-does-not-exist :create)
            (write-string clip out))
          (copy-to-clipboard clip)
          (echo "Clipped to ~a" (namestring *clips-file*))))))

;;; --- A summary by llm -------------------------------------------------------

(defvar *page-summaries* (make-hash-table :test #'equal)
  "Summaries made this session, by id: (title url markdown model).")

(define-internal-page page-summary (&key id)
    (:title "*Summary*")
  "A summary summarize-page made."
  (destructuring-bind (&optional title address markdown model) (gethash id *page-summaries*)
    (spinneret:with-html-string
      (:h1 (or title "Summary"))
      (:p (:a :href address address))
      (:raw (if markdown (pandoc markdown "-f" "gfm" "-t" "html") "<p>Gone: summarize the page again.</p>"))
      (:p (:small (format nil "By llm~@[ (~a)~]: a model's summary; check what matters." model))))))

(define-command-global summarize-page ()
  "Summarize the page with llm (your default model, or *summary-model*), in
a page of its own."
  (let* ((buffer (current-buffer))
         (text (page-markdown buffer))
         ;; Of a long page, the start (see *summary-max-chars*).
         (text (subseq text 0 (min (length text) *summary-max-chars*)))
         (id (princ-to-string (get-universal-time))))
    (echo "Summarizing ~a with llm..." (title buffer))
    (let ((summary
            (with-input-from-string (in text)
              (uiop:run-program
               (append (list "llm")
                       (when *summary-model* (list "-m" *summary-model*))
                       (list "-s" "Summarize this web page for a busy reader: 5 to 8 Markdown bullet points with the key facts, then one sentence on who it is for. Use only what the page says."))
               :input in :output :string :error-output :string))))
      (setf (gethash id *page-summaries*)
            (list (title buffer) (render-url (url buffer)) summary *summary-model*))
      (buffer-load-internal-page-focus 'page-summary :id id))))

;;; --- Tables and code blocks -------------------------------------------------

(defun page-folder (buffer)
  "~/Documents/Nyxt/<page title>/: for what a page gives several of."
  (merge-pathnames (format nil "~a/" (page-file-name (title buffer))) *page-tools-folder*))

(defun one-line (text)
  "TEXT with its runs of spaces and line breaks made one space: a cell or
a heading as the page shows it, not as its HTML was indented."
  (string-trim " " (str:join " " (str:split-omit-nulls " " (substitute #\Space #\Tab (substitute #\Space #\Newline text))))))

(defun csv-field (text)
  "TEXT as one CSV field: in quotes when it holds a comma or a quote."
  (let ((text (one-line text)))
    (if (find-if (lambda (c) (find c ",\"")) text)
        (format nil "\"~a\"" (str:replace-all "\"" "\"\"" text))
        text)))

(defun table-rows (table)
  "TABLE's rows, each the text of its cells."
  (loop for tr across (clss:select "tr" table)
        collect (loop for cell across (clss:select "th, td" tr)
                      collect (plump:text cell))))

(define-command-global save-page-tables ()
  "Save every table on the page as CSV, a file each, in
~/Documents/Nyxt/<page title>/table-N.csv, for a spreadsheet."
  (let* ((buffer (current-buffer))
         (tables (clss:select "table" (document-model buffer)))
         (folder (page-folder buffer)))
    (if (zerop (length tables))
        (echo "No tables on this page")
        (progn
          (ensure-directories-exist folder)
          (loop for table across tables
                for n from 1
                do (with-open-file (out (merge-pathnames (format nil "table-~d.csv" n) folder)
                                        :direction :output :if-exists :supersede)
                     (dolist (row (table-rows table))
                       (format out "~{~a~^,~}~%" (mapcar #'csv-field row)))))
          (echo "~d table~:p saved to ~a" (length tables) (namestring folder))))))

(defvar *code-extensions*
  '(("python" . "py") ("py" . "py") ("lisp" . "lisp") ("common-lisp" . "lisp") ("elisp" . "el")
    ("shell" . "sh") ("bash" . "sh") ("sh" . "sh") ("console" . "sh")
    ("javascript" . "js") ("js" . "js") ("typescript" . "ts") ("c" . "c") ("cpp" . "cpp")
    ("c++" . "cpp") ("rust" . "rs") ("go" . "go") ("ruby" . "rb") ("html" . "html")
    ("css" . "css") ("json" . "json") ("yaml" . "yaml") ("toml" . "toml") ("sql" . "sql")
    ("markdown" . "md") ("lua" . "lua") ("java" . "java") ("kotlin" . "kt"))
  "A file extension for the language a page names on a code block; .txt otherwise.")

(defclass code-block ()
  ((number :initarg :number :reader block-number)
   (text :initarg :text :reader block-text)
   (language :initarg :language :reader block-language))
  (:documentation "One <pre> of a page, in Ctrl+Space as its number, first line, language and length."))

(defmethod prompter:object-attributes ((block code-block) (source prompter:source))
  `(("Block" ,(block-number block))
    ("First line" ,(one-line (first (str:lines (block-text block)))))
    ("Language" ,(or (block-language block) ""))
    ("Lines" ,(length (str:lines (block-text block))))))

(defun code-blocks (buffer)
  "The page's code blocks (each <pre>, as code-block objects), with the
language a class like language-python names."
  (loop for pre across (clss:select "pre" (document-model buffer))
        for code = (let ((inner (clss:select "code" pre))) (if (plusp (length inner)) (elt inner 0) pre))
        for text = (string-right-trim '(#\Newline #\Space) (plump:text code))
        for classes = (format nil "~a ~a" (or (plump:attribute code "class") "") (or (plump:attribute pre "class") ""))
        for language = (loop for word in (str:split-omit-nulls " " classes)
                             for name = (cond ((str:starts-with-p "language-" word) (subseq word 9))
                                              ((str:starts-with-p "lang-" word) (subseq word 5)))
                             when name return (string-downcase name))
        for n from 1
        when (plusp (length text))
          collect (make-instance 'code-block :number n :text text :language language)))

(define-command-global save-page-code ()
  "Save every code block on the page as a file,
~/Documents/Nyxt/<page title>/code-N.EXT, the extension from the language
the page names (.txt when it names none)."
  (let* ((buffer (current-buffer))
         (blocks (code-blocks buffer))
         (folder (page-folder buffer)))
    (if (null blocks)
        (echo "No code blocks on this page")
        (progn
          (ensure-directories-exist folder)
          (dolist (block blocks)
            (let ((file (merge-pathnames
                         (format nil "code-~d.~a" (block-number block)
                                 (or (cdr (assoc (block-language block) *code-extensions* :test #'equal)) "txt"))
                         folder)))
              (alexandria:write-string-into-file (format nil "~a~%" (block-text block)) file :if-exists :supersede)))
          (echo "~d code block~:p saved to ~a" (length blocks) (namestring folder))))))

(define-command-global copy-page-code ()
  "Pick one of the page's code blocks and copy it, to paste into a
terminal or an editor."
  (let ((blocks (code-blocks (current-buffer))))
    (if (null blocks)
        (echo "No code blocks on this page")
        (let ((block (prompt1 :prompt "Copy which code block"
                              :sources (make-instance 'prompter:source
                                                      :name "Code blocks"
                                                      :constructor blocks))))
          (copy-code-block block)))))

(defun copy-code-block (block)
  (copy-to-clipboard (format nil "~a~%" (block-text block)))
  (echo "Block ~d copied (~d lines)" (block-number block) (length (str:lines (block-text block)))))

;;; --- Headings ---------------------------------------------------------------

(defclass heading ()
  ((index :initarg :index :reader heading-index)
   (level :initarg :level :reader heading-level)
   (text :initarg :text :reader heading-text))
  (:documentation "One heading of the page: its place among all of them (what the
page is scrolled to), its level and its words."))

(defmethod prompter:object-attributes ((heading heading) (source prompter:source))
  `(("Heading" ,(format nil "~a~a" (make-string (* 2 (1- (heading-level heading))) :initial-element #\Space) (heading-text heading)))
    ("Level" ,(format nil "h~d" (heading-level heading)))))

(defparameter *headings-selector* "h1, h2, h3, h4, h5, h6")

(defun page-headings (buffer)
  "The page's headings in its order, as heading objects."
  (loop for node across (clss:select *headings-selector* (document-model buffer))
        for i from 0
        collect (make-instance 'heading :index i
                                        :level (digit-char-p (char (plump:tag-name node) 1))
                                        :text (one-line (plump:text node)))))

(define-command-global jump-to-heading ()
  "Scroll the page to one of its headings, picked from a list of them."
  (let* ((buffer (current-buffer))
         (headings (page-headings buffer)))
    (if (null headings)
        (echo "No headings on this page")
        (let ((heading (prompt1 :prompt "Jump to heading"
                                :sources (make-instance 'prompter:source
                                                        :name "Headings"
                                                        :constructor headings))))
          (scroll-to-heading buffer heading)))))

(defun scroll-to-heading (buffer heading)
  ;; The same selector in the page: the Nth of Nyxt's parsed copy is the
  ;; Nth there.
  (ps-eval :buffer buffer
    (let ((all (ps:chain document (query-selector-all (ps:lisp *headings-selector*)))))
      (ps:chain (aref all (ps:lisp (heading-index heading))) (scroll-into-view))))
  (echo "~a" (heading-text heading)))

;;; --- Reader view ------------------------------------------------------------

(defvar *reader-pages* (make-hash-table :test #'equal)
  "Reader views made this session, by id: (title url markdown).")

(define-internal-page page-reader (&key id)
    (:title "*Reader*")
  "The page's article alone, as reader-view shows it."
  (destructuring-bind (&optional title address markdown) (gethash id *reader-pages*)
    (spinneret:with-html-string
      (:div :style "max-width: 42em; margin: 0 auto; line-height: 1.5"
            (:h1 (or title "Reader"))
            (:p (:small (:a :href address address)))
            (:raw (if markdown
                      (pandoc markdown "-f" "gfm" "-t" "html")
                      "<p>Gone: open the reader view again.</p>"))))))

(define-command-global reader-view ()
  "The page's article alone (no menus, no footers, when the page marks
its article), as a clean page in Nyxt's own style, links still working."
  (let* ((buffer (current-buffer))
         (id (princ-to-string (get-universal-time))))
    (setf (gethash id *reader-pages*)
          (list (title buffer) (render-url (url buffer)) (page-markdown buffer)))
    (buffer-load-internal-page-focus 'page-reader :id id)))

;;; --- Into your notes, and a project's log -----------------------------------

(defun notes-folder ()
  "The folder `note index` reads (folder= in ~/.config/vikix/notes), or nil."
  (let ((file (uiop:xdg-config-home "vikix/notes")))
    (when (uiop:file-exists-p file)
      (loop for line in (uiop:read-file-lines file)
            when (str:starts-with-p "folder=" line)
              return (let ((folder (string-trim " " (subseq line 7))))
                       (when (plusp (length folder))
                         (uiop:ensure-directory-pathname
                          (if (str:starts-with-p "~/" folder)
                              (merge-pathnames (subseq folder 2) (user-homedir-pathname))
                              folder))))))))

(define-command-global add-page-to-notes ()
  "Save the page as Markdown into your notes folder, Web/<page title>.md,
and index it, so `note ask` and `note find` see it beside your notes."
  (let ((folder (notes-folder)))
    (if (null folder)
        (echo "No notes folder yet: note index ~~/Notes, in a terminal")
        (let* ((buffer (current-buffer))
               (file (merge-pathnames (format nil "Web/~a.md" (page-file-name (title buffer))) folder))
               (markdown (format nil "# ~a~%~%Source: <~a>, saved ~a~%~%~a"
                                 (title buffer) (render-url (url buffer)) (today) (page-markdown buffer))))
          (ensure-directories-exist file)
          (alexandria:write-string-into-file markdown file :if-exists :supersede)
          (echo "Saved to ~a; indexing..." (namestring file))
          ;; The index takes a while (the model reads the page): a thread, so
          ;; Nyxt isn't held.
          (run-thread "note index"
            (handler-case
                (progn (uiop:run-program '("vikix" "notes" "index") :output nil :error-output nil)
                       (echo "~a is in your notes' index" (title buffer)))
              (error () (echo "Saved to ~a, but note index failed: run it in a terminal" (namestring file)))))))))

(defun selection-html (buffer)
  "What is selected in BUFFER, as HTML; an empty string when nothing is."
  (or (ps-eval :buffer buffer
        (let ((selection (ps:chain window (get-selection))))
          (if (> (ps:@ selection range-count) 0)
              (let ((div (ps:chain document (create-element "div"))))
                (ps:chain div (append-child (ps:chain selection (get-range-at 0)
                                                      (clone-contents))))
                (ps:@ div inner-h-t-m-l))
              "")))
      ""))

(defun selection-quote (buffer)
  "The selection as a Markdown quote (links made whole), or nil with
nothing selected."
  (let ((html (selection-html buffer)))
    (unless (string= (string-trim '(#\Space #\Newline) html) "")
      (let* ((html (plump:serialize (make-addresses-whole buffer (plump:parse html)) nil))
             (markdown (string-trim '(#\Newline #\Space)
                                    (pandoc html "-f" "html" "-t" "gfm-raw_html" "--wrap=none"))))
        (format nil "~{> ~a~^~%~}" (str:lines markdown))))))

(defun project-names ()
  "Your projects' names, as `vikix project list` gives them (its first word a line)."
  (let ((out (ignore-errors (uiop:run-program '("vikix" "project" "list") :output :string :error-output nil))))
    (when out
      (loop for line in (str:lines out)
            for name = (first (str:split-omit-nulls " " line))
            when name collect name))))

(define-command-global clip-to-project-log ()
  "Add the selected text as a quote, with the page's title and address, to
a project's log (vikix project log), the project picked from your list;
with nothing selected, the title and address alone: read today."
  (let* ((buffer (current-buffer))
         (names (project-names)))
    (if (null names)
        (echo "No projects: vikix project new NAME, in a terminal")
        (let* ((name (prompt1 :prompt "Into which project's log"
                              :sources (make-instance 'prompter:source
                                                      :name "Projects"
                                                      :constructor names)))
               (quote (selection-quote buffer)))
          (log-to-project buffer name quote)))))

(defun log-to-project (buffer name quote)
  "An entry in NAME's log: the page, and QUOTE (its selection) when there is one."
  (let ((entry (format nil "Read [~a](~a)~@[:~%~%~a~]" (title buffer) (render-url (url buffer)) quote)))
    (handler-case
        (progn
          (uiop:run-program (list "vikix" "project" "log" name entry "--vid") :output nil :error-output :string)
          (echo "Logged in ~a~@[, with the selection~]" name quote))
      (error (e) (echo "vikix project log failed: ~a" e)))))

A few things learned while writing it, in case you write your own:

  • remove-duplicates keeps the last copy of each unless given :from-end t: without it the pictures came out of order.
  • An empty result from clss:select is an empty vector, not nil, so (or (clss:select "main" dom) ...) never falls through: test its length.
  • show-url-qrcode isn’t in nyxt-user: from Lisp it’s nyxt/mode/document:show-url-qrcode; from Ctrl+Space, its name is enough.
  • llm takes the Anthropic key from the environment. A Nyxt started from the desktop has it.
  • A class’s reader can’t be called number or text: those are Common Lisp’s own symbols, and SBCL’s package lock refuses a function on them. Hence block-number, heading-text.
  • Ctrl+Space shows an object by prompter:object-attributes: a method on your class gives it columns (the code blocks’ first line, language and length), and the pick comes back as the object.

12.8 Ideas for your own

Some directions, from small to big, none of them built yet:


12.9 When Nyxt misbehaves

(handler-case (load (merge-pathnames "vikix/nyxt/vikix.lisp" (uiop:xdg-data-home)))
  (file-error () nil))

12.10 For the curious: how Vikix’s part works

All of it is config/nyxt/vikix.lisp in the checkout, one file of Lisp loaded into Nyxt: