* [PATCH] Async session eval (2nd attempt)
@ 2020-10-25 18:54 Jack Kamm
2020-10-26 2:23 ` stardiviner
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Jack Kamm @ 2020-10-25 18:54 UTC (permalink / raw)
To: emacs-orgmode
[-- Attachment #1: Type: text/plain, Size: 1980 bytes --]
This patch adds asynchronous evaluation for session blocks in
Python. It also adds functionality to implement async session eval for
other languages using ob-comint.el.
To test the attached patch, add ":async" to a Python session block
with a long computation (or "time.sleep") in it. Upon evaluation, your
Emacs won't freeze to wait for the result -- instead, a placeholder
will be inserted, and replaced with the true result when it's ready.
I'll note how this is different from some related projects. ob-async
implements asynchronous evaluation for Babel, but it doesn't work with
sessions. emacs-jupyter, ein, and ob-ipython all implement
asynchronous session evaluation, but only for Jupyter kernels. Jupyter
is great for some cases, but sometimes I prefer to use the built-in
org-babel languages without jupyter.
The new functionality is mainly implemented in
`org-babel-comint-async-filter', which I've defined in ob-comint.el,
and added as a hook to `comint-output-filter-functions'. Whenever new
output is added to the comint buffer, the filter scans for an
indicator token (this is inspired by
`org-babel-comint-with-output'). Upon encountering the token, the
filter uses a regular expression to extract a UUID or temp-file
associated with the result, then searches for the appropriate location
to add the result to.
This is my 2nd attempt at this patch [0]. I have also ported it to an
external package [1], but would like to have this functionality in Org
proper, to permit better code reuse between async and sync
implementations. The external package also includes an R
implementation that I regularly use, as well as a Ruby implementation,
but I've left these out to keep this initial patch smaller, and also I
need to confirm copyright assignment on the Ruby implementation which
was externally contributed.
[0] https://orgmode.org/list/87muj04xim.fsf@jaheira.i-did-not-set--mail-host-address--so-tickle-me/
[1] https://github.com/jackkamm/ob-session-async
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-comint.el-ob-python.el-Async-session-evaluation.patch --]
[-- Type: text/x-patch, Size: 17119 bytes --]
From 8b7695a148d1831c916737650e115833cb7fc752 Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Sun, 25 Oct 2020 11:40:10 -0700
Subject: [PATCH] ob-comint.el, ob-python.el: Async session evaluation
Adds functionality to ob-comint.el to implement async session eval on
a per-language basis. Adds a reference implementation for ob-python.
* lisp/ob-comint.el (org-babel-comint-with-output): Remove comment.
(org-babel-comint-async-indicator, org-babel-comint-async-buffers,
org-babel-comint-async-file-callback,
org-babel-comint-async-chunk-callback,
org-babel-comint-async-dangling): Add buffer-local variables used for
async comint evaluation.
(org-babel-comint-use-async): Add function to determine whether block
should be evaluated asynchronously.
(org-babel-comint-async-filter): Add filter function to attach to
comint-output-filter-functions for babel async eval.
(org-babel-comint-async-register): Add function to setup buffer
variables and hooks for session eval.
(org-babel-comint-async-delete-dangling-and-eval): Add helper function
for async session eval.
* lisp/ob-python.el (org-babel-execute:python): Check for async header
argument.
(org-babel-python-evaluate): Check whether to use async evaluation.
(org-babel-python-async-indicator): Add constant for indicating the
start/end of async evaluations.
(org-babel-python-async-evaluate-session): Add function for Python
async eval.
*
testing/lisp/test-ob-python.el (test-ob-python/async-simple-session-output):
Unit test for Python async session eval.
(test-ob-python/async-named-output): Unit test that Python async eval
can replace named output.
(test-ob-python/async-output-drawer): Unit test that Python async eval
works with drawer results.
---
etc/ORG-NEWS | 15 +++
lisp/ob-comint.el | 172 +++++++++++++++++++++++++++++++--
lisp/ob-python.el | 56 ++++++++++-
testing/lisp/test-ob-python.el | 61 ++++++++++++
4 files changed, 294 insertions(+), 10 deletions(-)
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 7f935bf52..9d5fbbe30 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -88,6 +88,21 @@ package, to convert pandas Dataframes into orgmode tables:
| 2 | 3 | 6 |
#+end_src
+*** Async session evaluation
+
+The =:async= header argument can be used for asynchronous evaluation
+in session blocks for certain languages.
+
+Currently, async evaluation is supported in Python. There is also
+functionality to implement async evaluation in other languages that
+use comint, but this needs to be done on a per-language basis.
+
+By default, async evaluation is disabled unless the =:async= header
+argument is present. You can also set =:async no= to force it off
+(for example if you've set =:async= in a property drawer).
+
+Async evaluation is disabled during export.
+
* Version 9.4
** Incompatible changes
*** Possibly broken internal file links: please check and fix
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index d3484bb7c..591754dac 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -94,12 +94,7 @@ (defmacro org-babel-comint-with-output (meta &rest body)
(regexp-quote ,eoe-indicator) nil t)
(re-search-forward
comint-prompt-regexp nil t)))))
- (accept-process-output (get-buffer-process (current-buffer)))
- ;; thought the following this would allow async
- ;; background running, but I was wrong...
- ;; (run-with-timer .5 .5 'accept-process-output
- ;; (get-buffer-process (current-buffer)))
- )
+ (accept-process-output (get-buffer-process (current-buffer))))
;; replace cut dangling text
(goto-char (process-mark (get-buffer-process (current-buffer))))
(insert dangling-text)
@@ -149,6 +144,171 @@ (defun org-babel-comint-eval-invisibly-and-wait-for-file
(if (= (aref string (1- (length string))) ?\n) string (concat string "\n")))
(while (not (file-exists-p file)) (sit-for (or period 0.25))))
+
+;; Async evaluation
+
+(defvar-local org-babel-comint-async-indicator nil
+ "Regular expression that `org-babel-comint-async-filter' scans for.
+It should have 2 parenthesized expressions,
+e.g. \"org_babel_async_\\(start\\|end\\|file\\)_\\(.*\\)\". The
+first parenthesized expression determines whether the token is
+delimiting a result block, or whether the result is in a file. If
+delimiting a block, the second expression gives a UUID for the
+location to insert the result. Otherwise, the result is in a tmp
+file, and the second expression gives the file name.")
+
+(defvar-local org-babel-comint-async-buffers nil
+ "List of org-mode buffers to check for Babel async output results.")
+
+(defvar-local org-babel-comint-async-file-callback nil
+ "Callback to clean and insert Babel async results from a temp file.
+The callback function takes two arguments: the alist of params of the Babel
+source block, and the name of the temp file.")
+
+(defvar-local org-babel-comint-async-chunk-callback nil
+ "Callback function to clean Babel async output results before insertion.
+Its single argument is a string consisting of output from the
+comint process. It should return a string that will be be passed
+to `org-babel-insert-result'.")
+
+(defvar-local org-babel-comint-async-dangling nil
+ "Dangling piece of the last process output, in case
+`org-babel-comint-async-indicator' is spread across multiple
+comint outputs due to buffering.")
+
+(defun org-babel-comint-use-async (params)
+ "Determine whether to use session async evaluation.
+PARAMS are the header arguments as passed to
+`org-babel-execute:lang'."
+ (let ((async (assq :async params))
+ (session (assq :session params)))
+ (and async
+ (not org-babel-exp-reference-buffer)
+ (not (equal (cdr async) "no"))
+ (not (equal (cdr session) "none")))))
+
+(defun org-babel-comint-async-filter (string)
+ "Captures Babel async output from comint buffer back to org-mode buffers.
+This function is added as a hook to `comint-output-filter-functions'.
+STRING contains the output originally inserted into the comint buffer."
+ ;; Remove outdated org-mode buffers
+ (setq org-babel-comint-async-buffers
+ (cl-loop for buf in org-babel-comint-async-buffers
+ if (buffer-live-p buf)
+ collect buf))
+ (let* ((indicator org-babel-comint-async-indicator)
+ (org-buffers org-babel-comint-async-buffers)
+ (file-callback org-babel-comint-async-file-callback)
+ (combined-string (concat org-babel-comint-async-dangling string))
+ (new-dangling combined-string)
+ ;; list of UUID's matched by `org-babel-comint-async-indicator'
+ uuid-list)
+ (with-temp-buffer
+ (insert combined-string)
+ (goto-char (point-min))
+ (while (re-search-forward indicator nil t)
+ ;; update dangling
+ (setq new-dangling (buffer-substring (point) (point-max)))
+ (cond ((equal (match-string 1) "end")
+ ;; save UUID for insertion later
+ (push (match-string 2) uuid-list))
+ ((equal (match-string 1) "file")
+ ;; insert results from tmp-file
+ (let ((tmp-file (match-string 2)))
+ (cl-loop for buf in org-buffers
+ until
+ (with-current-buffer buf
+ (save-excursion
+ (goto-char (point-min))
+ (when (search-forward tmp-file nil t)
+ (org-babel-previous-src-block)
+ (let* ((info (org-babel-get-src-block-info))
+ (params (nth 2 info))
+ (result-params
+ (cdr (assq :result-params params))))
+ (org-babel-insert-result
+ (funcall file-callback
+ (nth
+ 2 (org-babel-get-src-block-info))
+ tmp-file)
+ result-params info))
+ t))))))))
+ ;; Truncate dangling to only the most recent output
+ (when (> (length new-dangling) (length string))
+ (setq new-dangling string)))
+ (setq-local org-babel-comint-async-dangling new-dangling)
+ (when uuid-list
+ ;; Search for results in the comint buffer
+ (save-excursion
+ (goto-char (point-max))
+ (while uuid-list
+ (re-search-backward indicator)
+ (when (equal (match-string 1) "end")
+ (let* ((uuid (match-string-no-properties 2))
+ (res-str-raw
+ (buffer-substring
+ ;; move point to beginning of indicator
+ (- (match-beginning 0) 1)
+ ;; find the matching start indicator
+ (cl-loop for pos = (re-search-backward indicator)
+ until (and (equal (match-string 1) "start")
+ (equal (match-string 2) uuid))
+ finally return (+ 1 (match-end 0)))))
+ ;; Apply callback to clean up the result
+ (res-str (funcall org-babel-comint-async-chunk-callback
+ res-str-raw)))
+ ;; Search for uuid in associated org-buffers to insert results
+ (cl-loop for buf in org-buffers
+ until (with-current-buffer buf
+ (save-excursion
+ (goto-char (point-min))
+ (when (search-forward uuid nil t)
+ (org-babel-previous-src-block)
+ (let* ((info (org-babel-get-src-block-info))
+ (params (nth 2 info))
+ (result-params
+ (cdr (assq :result-params params))))
+ (org-babel-insert-result
+ res-str result-params info))
+ t))))
+ ;; Remove uuid from the list to search for
+ (setq uuid-list (delete uuid uuid-list)))))))))
+
+(defun org-babel-comint-async-register
+ (session-buffer org-buffer indicator-regexp
+ chunk-callback file-callback)
+ "Sets local org-babel-comint-async variables in SESSION-BUFFER.
+ORG-BUFFER is added to `org-babel-comint-async-buffers' if not
+present. `org-babel-comint-async-indicator',
+`org-babel-comint-async-chunk-callback', and
+`org-babel-comint-async-file-callback' are set to
+INDICATOR-REGEXP, CHUNK-CALLBACK, and FILE-CALLBACK
+respectively."
+ (org-babel-comint-in-buffer session-buffer
+ (setq org-babel-comint-async-indicator indicator-regexp
+ org-babel-comint-async-chunk-callback chunk-callback
+ org-babel-comint-async-file-callback file-callback)
+ (unless (memq org-buffer org-babel-comint-async-buffers)
+ (setq org-babel-comint-async-buffers
+ (cons org-buffer org-babel-comint-async-buffers)))
+ (add-hook 'comint-output-filter-functions
+ 'org-babel-comint-async-filter nil t)))
+
+(defmacro org-babel-comint-async-delete-dangling-and-eval
+ (session-buffer &rest body)
+ "Remove dangling text in SESSION-BUFFER and evaluate BODY.
+This is analogous to `org-babel-comint-with-output', but meant
+for asynchronous output, and much shorter because inserting the
+result is delegated to `org-babel-comint-async-filter'."
+ (declare (indent 1))
+ `(org-babel-comint-in-buffer ,session-buffer
+ (goto-char (process-mark (get-buffer-process (current-buffer))))
+ (delete-region (point) (point-max))
+ ,@body))
+(def-edebug-spec org-babel-comint-async-with-output (sexp body))
+
(provide 'ob-comint)
+
+
;;; ob-comint.el ends here
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 6752adc17..e26c34b64 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -84,6 +84,7 @@ (defun org-babel-execute:python (body params)
(return-val (when (eq result-type 'value)
(cdr (assq :return params))))
(preamble (cdr (assq :preamble params)))
+ (async (org-babel-comint-use-async params))
(full-body
(concat
(org-babel-expand-body:generic
@@ -92,7 +93,8 @@ (defun org-babel-execute:python (body params)
(when return-val
(format (if session "\n%s" "\nreturn %s") return-val))))
(result (org-babel-python-evaluate
- session full-body result-type result-params preamble)))
+ session full-body result-type
+ result-params preamble async)))
(org-babel-reassemble-table
result
(org-babel-pick-name (cdr (assq :colname-names params))
@@ -278,11 +280,14 @@ (defun org-babel-python-format-session-value
(if (member "pp" result-params) "True" "False")))
(defun org-babel-python-evaluate
- (session body &optional result-type result-params preamble)
+ (session body &optional result-type result-params preamble async)
"Evaluate BODY as Python code."
(if session
- (org-babel-python-evaluate-session
- session body result-type result-params)
+ (if async
+ (org-babel-python-async-evaluate-session
+ session body result-type result-params)
+ (org-babel-python-evaluate-session
+ session body result-type result-params))
(org-babel-python-evaluate-external-process
body result-type result-params preamble)))
@@ -391,6 +396,49 @@ (defun org-babel-python-read-string (string)
(substring string 1 -1)
string))
+;; Async session eval
+
+(defconst org-babel-python-async-indicator "print ('ob_comint_async_python_%s_%s')")
+
+(defun org-babel-python-async-value-callback (params tmp-file)
+ (let ((result-params (cdr (assq :result-params params)))
+ (results (org-babel-eval-read-file tmp-file)))
+ (org-babel-result-cond result-params
+ results
+ (org-babel-python-table-or-string results))))
+
+(defun org-babel-python-async-evaluate-session
+ (session body &optional result-type result-params)
+ "Asynchronously evaluate BODY in SESSION.
+Returns a placeholder string for insertion, to later be replaced
+by `org-babel-comint-async-filter'."
+ (org-babel-comint-async-register
+ session (current-buffer)
+ "ob_comint_async_python_\\(.+\\)_\\(.+\\)"
+ 'org-babel-chomp 'org-babel-python-async-value-callback)
+ (let ((python-shell-buffer-name (org-babel-python-without-earmuffs session)))
+ (pcase result-type
+ (`output
+ (let ((uuid (md5 (number-to-string (random 100000000)))))
+ (with-temp-buffer
+ (insert (format org-babel-python-async-indicator "start" uuid))
+ (insert "\n")
+ (insert body)
+ (insert "\n")
+ (insert (format org-babel-python-async-indicator "end" uuid))
+ (python-shell-send-buffer))
+ uuid))
+ (`value
+ (let ((tmp-results-file (org-babel-temp-file "python-"))
+ (tmp-src-file (org-babel-temp-file "python-")))
+ (with-temp-file tmp-src-file (insert body))
+ (with-temp-buffer
+ (insert (org-babel-python-format-session-value tmp-src-file tmp-results-file result-params))
+ (insert "\n")
+ (insert (format org-babel-python-async-indicator "file" tmp-results-file))
+ (python-shell-send-buffer))
+ tmp-results-file)))))
+
(provide 'ob-python)
;;; ob-python.el ends here
diff --git a/testing/lisp/test-ob-python.el b/testing/lisp/test-ob-python.el
index a2cc7b79c..0267678cd 100644
--- a/testing/lisp/test-ob-python.el
+++ b/testing/lisp/test-ob-python.el
@@ -207,6 +207,67 @@ (ert-deftest test-ob-python/session-value-sleep ()
#+end_src"
(org-babel-execute-src-block)))))
+(ert-deftest test-ob-python/async-simple-session-output ()
+ (let ((org-babel-temporary-directory "/tmp")
+ (org-confirm-babel-evaluate nil))
+ (org-test-with-temp-text
+ "#+begin_src python :session :async yes :results output
+import time
+time.sleep(.1)
+print('Yep!')
+#+end_src\n"
+ (should (let ((expected "Yep!"))
+ (and (not (string= expected (org-babel-execute-src-block)))
+ (string= expected
+ (progn
+ (sleep-for 0 200)
+ (goto-char (org-babel-where-is-src-block-result))
+ (org-babel-read-result)))))))))
+
+(ert-deftest test-ob-python/async-named-output ()
+ (let (org-confirm-babel-evaluate
+ (org-babel-temporary-directory "/tmp")
+ (src-block "#+begin_src python :async :session :results output
+print(\"Yep!\")
+#+end_src")
+ (results-before "
+
+#+NAME: foobar
+#+RESULTS:
+: Nope!")
+ (results-after "
+
+#+NAME: foobar
+#+RESULTS:
+: Yep!
+"))
+ (org-test-with-temp-text
+ (concat src-block results-before)
+ (should (progn (org-babel-execute-src-block)
+ (sleep-for 0 200)
+ (string= (concat src-block results-after)
+ (buffer-string)))))))
+
+(ert-deftest test-ob-python/async-output-drawer ()
+ (let (org-confirm-babel-evaluate
+ (org-babel-temporary-directory "/tmp")
+ (src-block "#+begin_src python :async :session :results output drawer
+print(list(range(3)))
+#+end_src")
+ (result "
+
+#+RESULTS:
+:results:
+[0, 1, 2]
+:end:
+"))
+ (org-test-with-temp-text
+ src-block
+ (should (progn (org-babel-execute-src-block)
+ (sleep-for 0 200)
+ (string= (concat src-block result)
+ (buffer-string)))))))
+
(provide 'test-ob-python)
;;; test-ob-python.el ends here
--
2.28.0
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Async session eval (2nd attempt)
2020-10-25 18:54 [PATCH] Async session eval (2nd attempt) Jack Kamm
@ 2020-10-26 2:23 ` stardiviner
2020-10-26 9:46 ` Eric S Fraga
2020-11-09 4:09 ` Kyle Meyer
2021-01-03 8:51 ` TEC
2 siblings, 1 reply; 5+ messages in thread
From: stardiviner @ 2020-10-26 2:23 UTC (permalink / raw)
To: Jack Kamm; +Cc: emacs-orgmode
This really is an good idea. Some other Babel languages like Ruby, JavaScript
might benefit from this ob-comint async evaluation. Awesome!
Jack Kamm <jackkamm@gmail.com> writes:
> This patch adds asynchronous evaluation for session blocks in
> Python. It also adds functionality to implement async session eval for
> other languages using ob-comint.el.
>
> To test the attached patch, add ":async" to a Python session block
> with a long computation (or "time.sleep") in it. Upon evaluation, your
> Emacs won't freeze to wait for the result -- instead, a placeholder
> will be inserted, and replaced with the true result when it's ready.
>
> I'll note how this is different from some related projects. ob-async
> implements asynchronous evaluation for Babel, but it doesn't work with
> sessions. emacs-jupyter, ein, and ob-ipython all implement
> asynchronous session evaluation, but only for Jupyter kernels. Jupyter
> is great for some cases, but sometimes I prefer to use the built-in
> org-babel languages without jupyter.
>
> The new functionality is mainly implemented in
> `org-babel-comint-async-filter', which I've defined in ob-comint.el,
> and added as a hook to `comint-output-filter-functions'. Whenever new
> output is added to the comint buffer, the filter scans for an
> indicator token (this is inspired by
> `org-babel-comint-with-output'). Upon encountering the token, the
> filter uses a regular expression to extract a UUID or temp-file
> associated with the result, then searches for the appropriate location
> to add the result to.
>
> This is my 2nd attempt at this patch [0]. I have also ported it to an
> external package [1], but would like to have this functionality in Org
> proper, to permit better code reuse between async and sync
> implementations. The external package also includes an R
> implementation that I regularly use, as well as a Ruby implementation,
> but I've left these out to keep this initial patch smaller, and also I
> need to confirm copyright assignment on the Ruby implementation which
> was externally contributed.
>
> [0] https://orgmode.org/list/87muj04xim.fsf@jaheira.i-did-not-set--mail-host-address--so-tickle-me/
> [1] https://github.com/jackkamm/ob-session-async
>
> From 8b7695a148d1831c916737650e115833cb7fc752 Mon Sep 17 00:00:00 2001
> From: Jack Kamm <jackkamm@gmail.com>
> Date: Sun, 25 Oct 2020 11:40:10 -0700
> Subject: [PATCH] ob-comint.el, ob-python.el: Async session evaluation
>
> Adds functionality to ob-comint.el to implement async session eval on
> a per-language basis. Adds a reference implementation for ob-python.
>
> * lisp/ob-comint.el (org-babel-comint-with-output): Remove comment.
> (org-babel-comint-async-indicator, org-babel-comint-async-buffers,
> org-babel-comint-async-file-callback,
> org-babel-comint-async-chunk-callback,
> org-babel-comint-async-dangling): Add buffer-local variables used for
> async comint evaluation.
> (org-babel-comint-use-async): Add function to determine whether block
> should be evaluated asynchronously.
> (org-babel-comint-async-filter): Add filter function to attach to
> comint-output-filter-functions for babel async eval.
> (org-babel-comint-async-register): Add function to setup buffer
> variables and hooks for session eval.
> (org-babel-comint-async-delete-dangling-and-eval): Add helper function
> for async session eval.
>
> * lisp/ob-python.el (org-babel-execute:python): Check for async header
> argument.
> (org-babel-python-evaluate): Check whether to use async evaluation.
> (org-babel-python-async-indicator): Add constant for indicating the
> start/end of async evaluations.
> (org-babel-python-async-evaluate-session): Add function for Python
> async eval.
>
> *
> testing/lisp/test-ob-python.el (test-ob-python/async-simple-session-output):
> Unit test for Python async session eval.
> (test-ob-python/async-named-output): Unit test that Python async eval
> can replace named output.
> (test-ob-python/async-output-drawer): Unit test that Python async eval
> works with drawer results.
> ---
> etc/ORG-NEWS | 15 +++
> lisp/ob-comint.el | 172 +++++++++++++++++++++++++++++++--
> lisp/ob-python.el | 56 ++++++++++-
> testing/lisp/test-ob-python.el | 61 ++++++++++++
> 4 files changed, 294 insertions(+), 10 deletions(-)
>
> diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
> index 7f935bf52..9d5fbbe30 100644
> --- a/etc/ORG-NEWS
> +++ b/etc/ORG-NEWS
> @@ -88,6 +88,21 @@ package, to convert pandas Dataframes into orgmode tables:
> | 2 | 3 | 6 |
> #+end_src
>
> +*** Async session evaluation
> +
> +The =:async= header argument can be used for asynchronous evaluation
> +in session blocks for certain languages.
> +
> +Currently, async evaluation is supported in Python. There is also
> +functionality to implement async evaluation in other languages that
> +use comint, but this needs to be done on a per-language basis.
> +
> +By default, async evaluation is disabled unless the =:async= header
> +argument is present. You can also set =:async no= to force it off
> +(for example if you've set =:async= in a property drawer).
> +
> +Async evaluation is disabled during export.
> +
> * Version 9.4
> ** Incompatible changes
> *** Possibly broken internal file links: please check and fix
> diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
> index d3484bb7c..591754dac 100644
> --- a/lisp/ob-comint.el
> +++ b/lisp/ob-comint.el
> @@ -94,12 +94,7 @@ (defmacro org-babel-comint-with-output (meta &rest body)
> (regexp-quote ,eoe-indicator) nil t)
> (re-search-forward
> comint-prompt-regexp nil t)))))
> - (accept-process-output (get-buffer-process (current-buffer)))
> - ;; thought the following this would allow async
> - ;; background running, but I was wrong...
> - ;; (run-with-timer .5 .5 'accept-process-output
> - ;; (get-buffer-process (current-buffer)))
> - )
> + (accept-process-output (get-buffer-process (current-buffer))))
> ;; replace cut dangling text
> (goto-char (process-mark (get-buffer-process (current-buffer))))
> (insert dangling-text)
> @@ -149,6 +144,171 @@ (defun org-babel-comint-eval-invisibly-and-wait-for-file
> (if (= (aref string (1- (length string))) ?\n) string (concat string "\n")))
> (while (not (file-exists-p file)) (sit-for (or period 0.25))))
>
> +
> +;; Async evaluation
> +
> +(defvar-local org-babel-comint-async-indicator nil
> + "Regular expression that `org-babel-comint-async-filter' scans for.
> +It should have 2 parenthesized expressions,
> +e.g. \"org_babel_async_\\(start\\|end\\|file\\)_\\(.*\\)\". The
> +first parenthesized expression determines whether the token is
> +delimiting a result block, or whether the result is in a file. If
> +delimiting a block, the second expression gives a UUID for the
> +location to insert the result. Otherwise, the result is in a tmp
> +file, and the second expression gives the file name.")
> +
> +(defvar-local org-babel-comint-async-buffers nil
> + "List of org-mode buffers to check for Babel async output results.")
> +
> +(defvar-local org-babel-comint-async-file-callback nil
> + "Callback to clean and insert Babel async results from a temp file.
> +The callback function takes two arguments: the alist of params of the Babel
> +source block, and the name of the temp file.")
> +
> +(defvar-local org-babel-comint-async-chunk-callback nil
> + "Callback function to clean Babel async output results before insertion.
> +Its single argument is a string consisting of output from the
> +comint process. It should return a string that will be be passed
> +to `org-babel-insert-result'.")
> +
> +(defvar-local org-babel-comint-async-dangling nil
> + "Dangling piece of the last process output, in case
> +`org-babel-comint-async-indicator' is spread across multiple
> +comint outputs due to buffering.")
> +
> +(defun org-babel-comint-use-async (params)
> + "Determine whether to use session async evaluation.
> +PARAMS are the header arguments as passed to
> +`org-babel-execute:lang'."
> + (let ((async (assq :async params))
> + (session (assq :session params)))
> + (and async
> + (not org-babel-exp-reference-buffer)
> + (not (equal (cdr async) "no"))
> + (not (equal (cdr session) "none")))))
> +
> +(defun org-babel-comint-async-filter (string)
> + "Captures Babel async output from comint buffer back to org-mode buffers.
> +This function is added as a hook to `comint-output-filter-functions'.
> +STRING contains the output originally inserted into the comint buffer."
> + ;; Remove outdated org-mode buffers
> + (setq org-babel-comint-async-buffers
> + (cl-loop for buf in org-babel-comint-async-buffers
> + if (buffer-live-p buf)
> + collect buf))
> + (let* ((indicator org-babel-comint-async-indicator)
> + (org-buffers org-babel-comint-async-buffers)
> + (file-callback org-babel-comint-async-file-callback)
> + (combined-string (concat org-babel-comint-async-dangling string))
> + (new-dangling combined-string)
> + ;; list of UUID's matched by `org-babel-comint-async-indicator'
> + uuid-list)
> + (with-temp-buffer
> + (insert combined-string)
> + (goto-char (point-min))
> + (while (re-search-forward indicator nil t)
> + ;; update dangling
> + (setq new-dangling (buffer-substring (point) (point-max)))
> + (cond ((equal (match-string 1) "end")
> + ;; save UUID for insertion later
> + (push (match-string 2) uuid-list))
> + ((equal (match-string 1) "file")
> + ;; insert results from tmp-file
> + (let ((tmp-file (match-string 2)))
> + (cl-loop for buf in org-buffers
> + until
> + (with-current-buffer buf
> + (save-excursion
> + (goto-char (point-min))
> + (when (search-forward tmp-file nil t)
> + (org-babel-previous-src-block)
> + (let* ((info (org-babel-get-src-block-info))
> + (params (nth 2 info))
> + (result-params
> + (cdr (assq :result-params params))))
> + (org-babel-insert-result
> + (funcall file-callback
> + (nth
> + 2 (org-babel-get-src-block-info))
> + tmp-file)
> + result-params info))
> + t))))))))
> + ;; Truncate dangling to only the most recent output
> + (when (> (length new-dangling) (length string))
> + (setq new-dangling string)))
> + (setq-local org-babel-comint-async-dangling new-dangling)
> + (when uuid-list
> + ;; Search for results in the comint buffer
> + (save-excursion
> + (goto-char (point-max))
> + (while uuid-list
> + (re-search-backward indicator)
> + (when (equal (match-string 1) "end")
> + (let* ((uuid (match-string-no-properties 2))
> + (res-str-raw
> + (buffer-substring
> + ;; move point to beginning of indicator
> + (- (match-beginning 0) 1)
> + ;; find the matching start indicator
> + (cl-loop for pos = (re-search-backward indicator)
> + until (and (equal (match-string 1) "start")
> + (equal (match-string 2) uuid))
> + finally return (+ 1 (match-end 0)))))
> + ;; Apply callback to clean up the result
> + (res-str (funcall org-babel-comint-async-chunk-callback
> + res-str-raw)))
> + ;; Search for uuid in associated org-buffers to insert results
> + (cl-loop for buf in org-buffers
> + until (with-current-buffer buf
> + (save-excursion
> + (goto-char (point-min))
> + (when (search-forward uuid nil t)
> + (org-babel-previous-src-block)
> + (let* ((info (org-babel-get-src-block-info))
> + (params (nth 2 info))
> + (result-params
> + (cdr (assq :result-params params))))
> + (org-babel-insert-result
> + res-str result-params info))
> + t))))
> + ;; Remove uuid from the list to search for
> + (setq uuid-list (delete uuid uuid-list)))))))))
> +
> +(defun org-babel-comint-async-register
> + (session-buffer org-buffer indicator-regexp
> + chunk-callback file-callback)
> + "Sets local org-babel-comint-async variables in SESSION-BUFFER.
> +ORG-BUFFER is added to `org-babel-comint-async-buffers' if not
> +present. `org-babel-comint-async-indicator',
> +`org-babel-comint-async-chunk-callback', and
> +`org-babel-comint-async-file-callback' are set to
> +INDICATOR-REGEXP, CHUNK-CALLBACK, and FILE-CALLBACK
> +respectively."
> + (org-babel-comint-in-buffer session-buffer
> + (setq org-babel-comint-async-indicator indicator-regexp
> + org-babel-comint-async-chunk-callback chunk-callback
> + org-babel-comint-async-file-callback file-callback)
> + (unless (memq org-buffer org-babel-comint-async-buffers)
> + (setq org-babel-comint-async-buffers
> + (cons org-buffer org-babel-comint-async-buffers)))
> + (add-hook 'comint-output-filter-functions
> + 'org-babel-comint-async-filter nil t)))
> +
> +(defmacro org-babel-comint-async-delete-dangling-and-eval
> + (session-buffer &rest body)
> + "Remove dangling text in SESSION-BUFFER and evaluate BODY.
> +This is analogous to `org-babel-comint-with-output', but meant
> +for asynchronous output, and much shorter because inserting the
> +result is delegated to `org-babel-comint-async-filter'."
> + (declare (indent 1))
> + `(org-babel-comint-in-buffer ,session-buffer
> + (goto-char (process-mark (get-buffer-process (current-buffer))))
> + (delete-region (point) (point-max))
> + ,@body))
> +(def-edebug-spec org-babel-comint-async-with-output (sexp body))
> +
> (provide 'ob-comint)
>
> +
> +
> ;;; ob-comint.el ends here
> diff --git a/lisp/ob-python.el b/lisp/ob-python.el
> index 6752adc17..e26c34b64 100644
> --- a/lisp/ob-python.el
> +++ b/lisp/ob-python.el
> @@ -84,6 +84,7 @@ (defun org-babel-execute:python (body params)
> (return-val (when (eq result-type 'value)
> (cdr (assq :return params))))
> (preamble (cdr (assq :preamble params)))
> + (async (org-babel-comint-use-async params))
> (full-body
> (concat
> (org-babel-expand-body:generic
> @@ -92,7 +93,8 @@ (defun org-babel-execute:python (body params)
> (when return-val
> (format (if session "\n%s" "\nreturn %s") return-val))))
> (result (org-babel-python-evaluate
> - session full-body result-type result-params preamble)))
> + session full-body result-type
> + result-params preamble async)))
> (org-babel-reassemble-table
> result
> (org-babel-pick-name (cdr (assq :colname-names params))
> @@ -278,11 +280,14 @@ (defun org-babel-python-format-session-value
> (if (member "pp" result-params) "True" "False")))
>
> (defun org-babel-python-evaluate
> - (session body &optional result-type result-params preamble)
> + (session body &optional result-type result-params preamble async)
> "Evaluate BODY as Python code."
> (if session
> - (org-babel-python-evaluate-session
> - session body result-type result-params)
> + (if async
> + (org-babel-python-async-evaluate-session
> + session body result-type result-params)
> + (org-babel-python-evaluate-session
> + session body result-type result-params))
> (org-babel-python-evaluate-external-process
> body result-type result-params preamble)))
>
> @@ -391,6 +396,49 @@ (defun org-babel-python-read-string (string)
> (substring string 1 -1)
> string))
>
> +;; Async session eval
> +
> +(defconst org-babel-python-async-indicator "print ('ob_comint_async_python_%s_%s')")
> +
> +(defun org-babel-python-async-value-callback (params tmp-file)
> + (let ((result-params (cdr (assq :result-params params)))
> + (results (org-babel-eval-read-file tmp-file)))
> + (org-babel-result-cond result-params
> + results
> + (org-babel-python-table-or-string results))))
> +
> +(defun org-babel-python-async-evaluate-session
> + (session body &optional result-type result-params)
> + "Asynchronously evaluate BODY in SESSION.
> +Returns a placeholder string for insertion, to later be replaced
> +by `org-babel-comint-async-filter'."
> + (org-babel-comint-async-register
> + session (current-buffer)
> + "ob_comint_async_python_\\(.+\\)_\\(.+\\)"
> + 'org-babel-chomp 'org-babel-python-async-value-callback)
> + (let ((python-shell-buffer-name (org-babel-python-without-earmuffs session)))
> + (pcase result-type
> + (`output
> + (let ((uuid (md5 (number-to-string (random 100000000)))))
> + (with-temp-buffer
> + (insert (format org-babel-python-async-indicator "start" uuid))
> + (insert "\n")
> + (insert body)
> + (insert "\n")
> + (insert (format org-babel-python-async-indicator "end" uuid))
> + (python-shell-send-buffer))
> + uuid))
> + (`value
> + (let ((tmp-results-file (org-babel-temp-file "python-"))
> + (tmp-src-file (org-babel-temp-file "python-")))
> + (with-temp-file tmp-src-file (insert body))
> + (with-temp-buffer
> + (insert (org-babel-python-format-session-value tmp-src-file tmp-results-file result-params))
> + (insert "\n")
> + (insert (format org-babel-python-async-indicator "file" tmp-results-file))
> + (python-shell-send-buffer))
> + tmp-results-file)))))
> +
> (provide 'ob-python)
>
> ;;; ob-python.el ends here
> diff --git a/testing/lisp/test-ob-python.el b/testing/lisp/test-ob-python.el
> index a2cc7b79c..0267678cd 100644
> --- a/testing/lisp/test-ob-python.el
> +++ b/testing/lisp/test-ob-python.el
> @@ -207,6 +207,67 @@ (ert-deftest test-ob-python/session-value-sleep ()
> #+end_src"
> (org-babel-execute-src-block)))))
>
> +(ert-deftest test-ob-python/async-simple-session-output ()
> + (let ((org-babel-temporary-directory "/tmp")
> + (org-confirm-babel-evaluate nil))
> + (org-test-with-temp-text
> + "#+begin_src python :session :async yes :results output
> +import time
> +time.sleep(.1)
> +print('Yep!')
> +#+end_src\n"
> + (should (let ((expected "Yep!"))
> + (and (not (string= expected (org-babel-execute-src-block)))
> + (string= expected
> + (progn
> + (sleep-for 0 200)
> + (goto-char (org-babel-where-is-src-block-result))
> + (org-babel-read-result)))))))))
> +
> +(ert-deftest test-ob-python/async-named-output ()
> + (let (org-confirm-babel-evaluate
> + (org-babel-temporary-directory "/tmp")
> + (src-block "#+begin_src python :async :session :results output
> +print(\"Yep!\")
> +#+end_src")
> + (results-before "
> +
> +#+NAME: foobar
> +#+RESULTS:
> +: Nope!")
> + (results-after "
> +
> +#+NAME: foobar
> +#+RESULTS:
> +: Yep!
> +"))
> + (org-test-with-temp-text
> + (concat src-block results-before)
> + (should (progn (org-babel-execute-src-block)
> + (sleep-for 0 200)
> + (string= (concat src-block results-after)
> + (buffer-string)))))))
> +
> +(ert-deftest test-ob-python/async-output-drawer ()
> + (let (org-confirm-babel-evaluate
> + (org-babel-temporary-directory "/tmp")
> + (src-block "#+begin_src python :async :session :results output drawer
> +print(list(range(3)))
> +#+end_src")
> + (result "
> +
> +#+RESULTS:
> +:results:
> +[0, 1, 2]
> +:end:
> +"))
> + (org-test-with-temp-text
> + src-block
> + (should (progn (org-babel-execute-src-block)
> + (sleep-for 0 200)
> + (string= (concat src-block result)
> + (buffer-string)))))))
> +
> (provide 'test-ob-python)
>
> ;;; test-ob-python.el ends here
--
[ stardiviner ]
I try to make every word tell the meaning that I want to express.
Blog: https://stardiviner.github.io/
IRC(freenode): stardiviner, Matrix: stardiviner
GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Async session eval (2nd attempt)
2020-10-26 2:23 ` stardiviner
@ 2020-10-26 9:46 ` Eric S Fraga
0 siblings, 0 replies; 5+ messages in thread
From: Eric S Fraga @ 2020-10-26 9:46 UTC (permalink / raw)
To: stardiviner; +Cc: Jack Kamm, emacs-orgmode
On Monday, 26 Oct 2020 at 10:23, stardiviner wrote:
> This really is an good idea. Some other Babel languages like Ruby, JavaScript
> might benefit from this ob-comint async evaluation. Awesome!
Julia would benefit from this as well! My Julia jobs usually take
minutes if not hours. I usually tangle to run the code separately but
would love to simply C-c C-c and let it go.
--
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-61-ga88806.dirty
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Async session eval (2nd attempt)
2020-10-25 18:54 [PATCH] Async session eval (2nd attempt) Jack Kamm
2020-10-26 2:23 ` stardiviner
@ 2020-11-09 4:09 ` Kyle Meyer
2021-01-03 8:51 ` TEC
2 siblings, 0 replies; 5+ messages in thread
From: Kyle Meyer @ 2020-11-09 4:09 UTC (permalink / raw)
To: Jack Kamm; +Cc: emacs-orgmode
Jack Kamm writes:
> This patch adds asynchronous evaluation for session blocks in
> Python. It also adds functionality to implement async session eval for
> other languages using ob-comint.el.
>
> To test the attached patch, add ":async" to a Python session block
> with a long computation (or "time.sleep") in it. Upon evaluation, your
> Emacs won't freeze to wait for the result -- instead, a placeholder
> will be inserted, and replaced with the true result when it's ready.
>
> [...]
Thanks. Sounds exciting :)
So... have any Babel users on the list given this a spin? Any comments
or feedback?
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Async session eval (2nd attempt)
2020-10-25 18:54 [PATCH] Async session eval (2nd attempt) Jack Kamm
2020-10-26 2:23 ` stardiviner
2020-11-09 4:09 ` Kyle Meyer
@ 2021-01-03 8:51 ` TEC
2 siblings, 0 replies; 5+ messages in thread
From: TEC @ 2021-01-03 8:51 UTC (permalink / raw)
To: Jack Kamm; +Cc: emacs-orgmode
Hi Jack,
I love the look of this! Thanks for submitting a patch.
Sorry it's taken so long for someone to take a look at it, I think a lot
of the 'main' Org people have been pretty busy over the last few months.
I just tried to give this a shot.
First up, I had to remove the ORG-NEWS part of the patch to be able to
provide it. It would be nice if you could update the patch so this
applies cleanly.
#+begin_example
error: patch failed: etc/ORG-NEWS:88
error: etc/ORG-NEWS: patch does not apply
#+end_example
To test this, after applying your patch (with ORG-NEWS removed), I
started emacs -Q, loaded Org, and opened a new file.
I was initially unable to get this to seem to work, until I changed the
:results type to "output".
See a excerpt from my test file below:
----- excerpt start -----
#+begin_src python :async :session blah :results output
from time import sleep
a=2
sleep(2)
print("Hi")
#+end_src
#+RESULTS:
: Hi
#+begin_src python :async :session blah
return(a)
#+end_src
#+RESULTS:
: /tmp/babel-62cQRX/python-EfJ4o4
#+begin_src python :async :session blah :results output
print(a)
#+end_src
#+RESULTS:
: 2
----- excerpt end -----
I'm surprised this didn't work with the non-output block though.
Other than this, I'm rather happy to see that when I tried to execute
two long running blocks at once, the second one was not executed until
the first completed :)
Finally, I see that this requires :session to be set in order to work.
Might it be possible to have this work for non-session blocks too? It
seems odd that what I'd imagine is the harder case (session blocks) is
supported, but one-shot (non-session) blocks aren't.
Thanks again for your work, and I look forward to seeing what else you
have in the future!
--
Timothy
p.s. After this is merged, it would be great to see support for other
languages grow :)
Jack Kamm <jackkamm@gmail.com> writes:
> This patch adds asynchronous evaluation for session blocks in
> Python. It also adds functionality to implement async session eval for
> other languages using ob-comint.el.
>
> To test the attached patch, add ":async" to a Python session block
> with a long computation (or "time.sleep") in it. Upon evaluation, your
> Emacs won't freeze to wait for the result -- instead, a placeholder
> will be inserted, and replaced with the true result when it's ready.
>
> I'll note how this is different from some related projects. ob-async
> implements asynchronous evaluation for Babel, but it doesn't work with
> sessions. emacs-jupyter, ein, and ob-ipython all implement
> asynchronous session evaluation, but only for Jupyter kernels. Jupyter
> is great for some cases, but sometimes I prefer to use the built-in
> org-babel languages without jupyter.
>
> The new functionality is mainly implemented in
> `org-babel-comint-async-filter', which I've defined in ob-comint.el,
> and added as a hook to `comint-output-filter-functions'. Whenever new
> output is added to the comint buffer, the filter scans for an
> indicator token (this is inspired by
> `org-babel-comint-with-output'). Upon encountering the token, the
> filter uses a regular expression to extract a UUID or temp-file
> associated with the result, then searches for the appropriate location
> to add the result to.
>
> This is my 2nd attempt at this patch [0]. I have also ported it to an
> external package [1], but would like to have this functionality in Org
> proper, to permit better code reuse between async and sync
> implementations. The external package also includes an R
> implementation that I regularly use, as well as a Ruby implementation,
> but I've left these out to keep this initial patch smaller, and also I
> need to confirm copyright assignment on the Ruby implementation which
> was externally contributed.
>
> [0] https://orgmode.org/list/87muj04xim.fsf@jaheira.i-did-not-set--mail-host-address--so-tickle-me/
> [1] https://github.com/jackkamm/ob-session-async
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-01-03 9:43 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-25 18:54 [PATCH] Async session eval (2nd attempt) Jack Kamm
2020-10-26 2:23 ` stardiviner
2020-10-26 9:46 ` Eric S Fraga
2020-11-09 4:09 ` Kyle Meyer
2021-01-03 8:51 ` TEC
Org-mode mailing list
This inbox may be cloned and mirrored by anyone:
git clone --mirror https://orgmode.org/list/0 list/git/0.git
# If you have public-inbox 1.1+ installed, you may
# initialize and index your mirror using the following commands:
public-inbox-init -V2 list list/ https://orgmode.org/list \
emacs-orgmode@gnu.org
public-inbox-index list
Example config snippet for mirrors.
Newsgroups are available over NNTP:
nntp://news.yhetil.org/yhetil.emacs.orgmode
nntp://news.gmane.io/gmane.emacs.orgmode
AGPL code for this site: git clone https://public-inbox.org/public-inbox.git