[PATCH] Async session eval (2nd attempt)

[PATCH] Async session eval (2nd attempt)

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 1980 bytes --]

This patch adds asynchronous evaluation for session blocks in
Python. It also adds functionality to implement async session eval for
other languages using ob-comint.el.

To test the attached patch, add ":async" to a Python session block
with a long computation (or "time.sleep") in it. Upon evaluation, your
Emacs won't freeze to wait for the result -- instead, a placeholder
will be inserted, and replaced with the true result when it's ready.

I'll note how this is different from some related projects. ob-async
implements asynchronous evaluation for Babel, but it doesn't work with
sessions. emacs-jupyter, ein, and ob-ipython all implement
asynchronous session evaluation, but only for Jupyter kernels. Jupyter
is great for some cases, but sometimes I prefer to use the built-in
org-babel languages without jupyter.

The new functionality is mainly implemented in
`org-babel-comint-async-filter', which I've defined in ob-comint.el,
and added as a hook to `comint-output-filter-functions'.  Whenever new
output is added to the comint buffer, the filter scans for an
indicator token (this is inspired by
`org-babel-comint-with-output'). Upon encountering the token, the
filter uses a regular expression to extract a UUID or temp-file
associated with the result, then searches for the appropriate location
to add the result to.

This is my 2nd attempt at this patch [0]. I have also ported it to an
external package [1], but would like to have this functionality in Org
proper, to permit better code reuse between async and sync
implementations. The external package also includes an R
implementation that I regularly use, as well as a Ruby implementation,
but I've left these out to keep this initial patch smaller, and also I
need to confirm copyright assignment on the Ruby implementation which
was externally contributed.

[0] https://orgmode.org/list/87muj04xim.fsf@jaheira.i-did-not-set--mail-host-address--so-tickle-me/
[1] https://github.com/jackkamm/ob-session-async


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-comint.el-ob-python.el-Async-session-evaluation.patch --]
[-- Type: text/x-patch, Size: 17119 bytes --]

From 8b7695a148d1831c916737650e115833cb7fc752 Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Sun, 25 Oct 2020 11:40:10 -0700
Subject: [PATCH] ob-comint.el, ob-python.el: Async session evaluation

Adds functionality to ob-comint.el to implement async session eval on
a per-language basis.  Adds a reference implementation for ob-python.

* lisp/ob-comint.el (org-babel-comint-with-output): Remove comment.
(org-babel-comint-async-indicator, org-babel-comint-async-buffers,
org-babel-comint-async-file-callback,
org-babel-comint-async-chunk-callback,
org-babel-comint-async-dangling): Add buffer-local variables used for
async comint evaluation.
(org-babel-comint-use-async): Add function to determine whether block
should be evaluated asynchronously.
(org-babel-comint-async-filter): Add filter function to attach to
comint-output-filter-functions for babel async eval.
(org-babel-comint-async-register): Add function to setup buffer
variables and hooks for session eval.
(org-babel-comint-async-delete-dangling-and-eval): Add helper function
for async session eval.

* lisp/ob-python.el (org-babel-execute:python): Check for async header
argument.
(org-babel-python-evaluate): Check whether to use async evaluation.
(org-babel-python-async-indicator): Add constant for indicating the
start/end of async evaluations.
(org-babel-python-async-evaluate-session): Add function for Python
async eval.

*
testing/lisp/test-ob-python.el (test-ob-python/async-simple-session-output):
Unit test for Python async session eval.
(test-ob-python/async-named-output): Unit test that Python async eval
can replace named output.
(test-ob-python/async-output-drawer): Unit test that Python async eval
works with drawer results.
---
 etc/ORG-NEWS                   |  15 +++
 lisp/ob-comint.el              | 172 +++++++++++++++++++++++++++++++--
 lisp/ob-python.el              |  56 ++++++++++-
 testing/lisp/test-ob-python.el |  61 ++++++++++++
 4 files changed, 294 insertions(+), 10 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 7f935bf52..9d5fbbe30 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -88,6 +88,21 @@ package, to convert pandas Dataframes into orgmode tables:
 | 2 | 3 | 6 |
 #+end_src
 
+*** Async session evaluation
+
+The =:async= header argument can be used for asynchronous evaluation
+in session blocks for certain languages.
+
+Currently, async evaluation is supported in Python.  There is also
+functionality to implement async evaluation in other languages that
+use comint, but this needs to be done on a per-language basis.
+
+By default, async evaluation is disabled unless the =:async= header
+argument is present.  You can also set =:async no= to force it off
+(for example if you've set =:async= in a property drawer).
+
+Async evaluation is disabled during export.
+
 * Version 9.4
 ** Incompatible changes
 *** Possibly broken internal file links: please check and fix
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index d3484bb7c..591754dac 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -94,12 +94,7 @@ (defmacro org-babel-comint-with-output (meta &rest body)
 			       (regexp-quote ,eoe-indicator) nil t)
 			      (re-search-forward
 			       comint-prompt-regexp nil t)))))
-	   (accept-process-output (get-buffer-process (current-buffer)))
-	   ;; thought the following this would allow async
-	   ;; background running, but I was wrong...
-	   ;; (run-with-timer .5 .5 'accept-process-output
-	   ;; 		 (get-buffer-process (current-buffer)))
-	   )
+	   (accept-process-output (get-buffer-process (current-buffer))))
 	 ;; replace cut dangling text
 	 (goto-char (process-mark (get-buffer-process (current-buffer))))
 	 (insert dangling-text)
@@ -149,6 +144,171 @@ (defun org-babel-comint-eval-invisibly-and-wait-for-file
    (if (= (aref string (1- (length string))) ?\n) string (concat string "\n")))
   (while (not (file-exists-p file)) (sit-for (or period 0.25))))
 
+
+;; Async evaluation
+
+(defvar-local org-babel-comint-async-indicator nil
+  "Regular expression that `org-babel-comint-async-filter' scans for.
+It should have 2 parenthesized expressions,
+e.g. \"org_babel_async_\\(start\\|end\\|file\\)_\\(.*\\)\". The
+first parenthesized expression determines whether the token is
+delimiting a result block, or whether the result is in a file. If
+delimiting a block, the second expression gives a UUID for the
+location to insert the result. Otherwise, the result is in a tmp
+file, and the second expression gives the file name.")
+
+(defvar-local org-babel-comint-async-buffers nil
+  "List of org-mode buffers to check for Babel async output results.")
+
+(defvar-local org-babel-comint-async-file-callback nil
+  "Callback to clean and insert Babel async results from a temp file.
+The callback function takes two arguments: the alist of params of the Babel
+source block, and the name of the temp file.")
+
+(defvar-local org-babel-comint-async-chunk-callback nil
+  "Callback function to clean Babel async output results before insertion.
+Its single argument is a string consisting of output from the
+comint process. It should return a string that will be be passed
+to `org-babel-insert-result'.")
+
+(defvar-local org-babel-comint-async-dangling nil
+  "Dangling piece of the last process output, in case
+`org-babel-comint-async-indicator' is spread across multiple
+comint outputs due to buffering.")
+
+(defun org-babel-comint-use-async (params)
+  "Determine whether to use session async evaluation.
+PARAMS are the header arguments as passed to
+`org-babel-execute:lang'."
+  (let ((async (assq :async params))
+        (session (assq :session params)))
+    (and async
+	 (not org-babel-exp-reference-buffer)
+         (not (equal (cdr async) "no"))
+         (not (equal (cdr session) "none")))))
+
+(defun org-babel-comint-async-filter (string)
+  "Captures Babel async output from comint buffer back to org-mode buffers.
+This function is added as a hook to `comint-output-filter-functions'.
+STRING contains the output originally inserted into the comint buffer."
+  ;; Remove outdated org-mode buffers
+  (setq org-babel-comint-async-buffers
+	(cl-loop for buf in org-babel-comint-async-buffers
+	      if (buffer-live-p buf)
+	      collect buf))
+  (let* ((indicator org-babel-comint-async-indicator)
+	 (org-buffers org-babel-comint-async-buffers)
+	 (file-callback org-babel-comint-async-file-callback)
+	 (combined-string (concat org-babel-comint-async-dangling string))
+	 (new-dangling combined-string)
+	 ;; list of UUID's matched by `org-babel-comint-async-indicator'
+	 uuid-list)
+    (with-temp-buffer
+      (insert combined-string)
+      (goto-char (point-min))
+      (while (re-search-forward indicator nil t)
+	;; update dangling
+	(setq new-dangling (buffer-substring (point) (point-max)))
+	(cond ((equal (match-string 1) "end")
+	       ;; save UUID for insertion later
+	       (push (match-string 2) uuid-list))
+	      ((equal (match-string 1) "file")
+	       ;; insert results from tmp-file
+	       (let ((tmp-file (match-string 2)))
+		 (cl-loop for buf in org-buffers
+		       until
+		       (with-current-buffer buf
+			 (save-excursion
+			   (goto-char (point-min))
+			   (when (search-forward tmp-file nil t)
+			     (org-babel-previous-src-block)
+                             (let* ((info (org-babel-get-src-block-info))
+                                    (params (nth 2 info))
+                                    (result-params
+                                     (cdr (assq :result-params params))))
+                               (org-babel-insert-result
+                                 (funcall file-callback
+                                          (nth
+                                           2 (org-babel-get-src-block-info))
+                                          tmp-file)
+                                result-params info))
+			     t))))))))
+      ;; Truncate dangling to only the most recent output
+      (when (> (length new-dangling) (length string))
+	(setq new-dangling string)))
+    (setq-local org-babel-comint-async-dangling new-dangling)
+    (when uuid-list
+      ;; Search for results in the comint buffer
+      (save-excursion
+	(goto-char (point-max))
+	(while uuid-list
+	  (re-search-backward indicator)
+	  (when (equal (match-string 1) "end")
+	    (let* ((uuid (match-string-no-properties 2))
+		   (res-str-raw
+		    (buffer-substring
+		     ;; move point to beginning of indicator
+                     (- (match-beginning 0) 1)
+		     ;; find the matching start indicator
+		     (cl-loop for pos = (re-search-backward indicator)
+			   until (and (equal (match-string 1) "start")
+				      (equal (match-string 2) uuid))
+			   finally return (+ 1 (match-end 0)))))
+		   ;; Apply callback to clean up the result
+		   (res-str (funcall org-babel-comint-async-chunk-callback
+                                     res-str-raw)))
+	      ;; Search for uuid in associated org-buffers to insert results
+	      (cl-loop for buf in org-buffers
+		    until (with-current-buffer buf
+			    (save-excursion
+			      (goto-char (point-min))
+			      (when (search-forward uuid nil t)
+				(org-babel-previous-src-block)
+                                (let* ((info (org-babel-get-src-block-info))
+                                       (params (nth 2 info))
+                                       (result-params
+                                        (cdr (assq :result-params params))))
+				  (org-babel-insert-result
+                                   res-str result-params info))
+				t))))
+	      ;; Remove uuid from the list to search for
+	      (setq uuid-list (delete uuid uuid-list)))))))))
+
+(defun org-babel-comint-async-register
+    (session-buffer org-buffer indicator-regexp
+		    chunk-callback file-callback)
+  "Sets local org-babel-comint-async variables in SESSION-BUFFER.
+ORG-BUFFER is added to `org-babel-comint-async-buffers' if not
+present.  `org-babel-comint-async-indicator',
+`org-babel-comint-async-chunk-callback', and
+`org-babel-comint-async-file-callback' are set to
+INDICATOR-REGEXP, CHUNK-CALLBACK, and FILE-CALLBACK
+respectively."
+  (org-babel-comint-in-buffer session-buffer
+    (setq org-babel-comint-async-indicator indicator-regexp
+	  org-babel-comint-async-chunk-callback chunk-callback
+	  org-babel-comint-async-file-callback file-callback)
+    (unless (memq org-buffer org-babel-comint-async-buffers)
+      (setq org-babel-comint-async-buffers
+	    (cons org-buffer org-babel-comint-async-buffers)))
+    (add-hook 'comint-output-filter-functions
+	      'org-babel-comint-async-filter nil t)))
+
+(defmacro org-babel-comint-async-delete-dangling-and-eval
+    (session-buffer &rest body)
+  "Remove dangling text in SESSION-BUFFER and evaluate BODY.
+This is analogous to `org-babel-comint-with-output', but meant
+for asynchronous output, and much shorter because inserting the
+result is delegated to `org-babel-comint-async-filter'."
+  (declare (indent 1))
+  `(org-babel-comint-in-buffer ,session-buffer
+     (goto-char (process-mark (get-buffer-process (current-buffer))))
+     (delete-region (point) (point-max))
+     ,@body))
+(def-edebug-spec org-babel-comint-async-with-output (sexp body))
+
 (provide 'ob-comint)
 
+
+
 ;;; ob-comint.el ends here
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 6752adc17..e26c34b64 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -84,6 +84,7 @@ (defun org-babel-execute:python (body params)
 	 (return-val (when (eq result-type 'value)
 		       (cdr (assq :return params))))
 	 (preamble (cdr (assq :preamble params)))
+	 (async (org-babel-comint-use-async params))
          (full-body
 	  (concat
 	   (org-babel-expand-body:generic
@@ -92,7 +93,8 @@ (defun org-babel-execute:python (body params)
 	   (when return-val
 	     (format (if session "\n%s" "\nreturn %s") return-val))))
          (result (org-babel-python-evaluate
-		  session full-body result-type result-params preamble)))
+		  session full-body result-type
+		  result-params preamble async)))
     (org-babel-reassemble-table
      result
      (org-babel-pick-name (cdr (assq :colname-names params))
@@ -278,11 +280,14 @@ (defun org-babel-python-format-session-value
 	  (if (member "pp" result-params) "True" "False")))
 
 (defun org-babel-python-evaluate
-  (session body &optional result-type result-params preamble)
+  (session body &optional result-type result-params preamble async)
   "Evaluate BODY as Python code."
   (if session
-      (org-babel-python-evaluate-session
-       session body result-type result-params)
+      (if async
+	  (org-babel-python-async-evaluate-session
+	   session body result-type result-params)
+	(org-babel-python-evaluate-session
+	 session body result-type result-params))
     (org-babel-python-evaluate-external-process
      body result-type result-params preamble)))
 
@@ -391,6 +396,49 @@ (defun org-babel-python-read-string (string)
       (substring string 1 -1)
     string))
 
+;; Async session eval
+
+(defconst org-babel-python-async-indicator "print ('ob_comint_async_python_%s_%s')")
+
+(defun org-babel-python-async-value-callback (params tmp-file)
+  (let ((result-params (cdr (assq :result-params params)))
+	(results (org-babel-eval-read-file tmp-file)))
+    (org-babel-result-cond result-params
+      results
+      (org-babel-python-table-or-string results))))
+
+(defun org-babel-python-async-evaluate-session
+    (session body &optional result-type result-params)
+  "Asynchronously evaluate BODY in SESSION.
+Returns a placeholder string for insertion, to later be replaced
+by `org-babel-comint-async-filter'."
+  (org-babel-comint-async-register
+   session (current-buffer)
+   "ob_comint_async_python_\\(.+\\)_\\(.+\\)"
+   'org-babel-chomp 'org-babel-python-async-value-callback)
+  (let ((python-shell-buffer-name (org-babel-python-without-earmuffs session)))
+    (pcase result-type
+      (`output
+       (let ((uuid (md5 (number-to-string (random 100000000)))))
+         (with-temp-buffer
+           (insert (format org-babel-python-async-indicator "start" uuid))
+           (insert "\n")
+           (insert body)
+           (insert "\n")
+           (insert (format org-babel-python-async-indicator "end" uuid))
+           (python-shell-send-buffer))
+         uuid))
+      (`value
+       (let ((tmp-results-file (org-babel-temp-file "python-"))
+             (tmp-src-file (org-babel-temp-file "python-")))
+         (with-temp-file tmp-src-file (insert body))
+         (with-temp-buffer
+           (insert (org-babel-python-format-session-value tmp-src-file tmp-results-file result-params))
+           (insert "\n")
+           (insert (format org-babel-python-async-indicator "file" tmp-results-file))
+           (python-shell-send-buffer))
+         tmp-results-file)))))
+
 (provide 'ob-python)
 
 ;;; ob-python.el ends here
diff --git a/testing/lisp/test-ob-python.el b/testing/lisp/test-ob-python.el
index a2cc7b79c..0267678cd 100644
--- a/testing/lisp/test-ob-python.el
+++ b/testing/lisp/test-ob-python.el
@@ -207,6 +207,67 @@ (ert-deftest test-ob-python/session-value-sleep ()
 #+end_src"
 	    (org-babel-execute-src-block)))))
 
+(ert-deftest test-ob-python/async-simple-session-output ()
+  (let ((org-babel-temporary-directory "/tmp")
+        (org-confirm-babel-evaluate nil))
+    (org-test-with-temp-text
+     "#+begin_src python :session :async yes :results output
+import time
+time.sleep(.1)
+print('Yep!')
+#+end_src\n"
+     (should (let ((expected "Yep!"))
+	       (and (not (string= expected (org-babel-execute-src-block)))
+		    (string= expected
+			     (progn
+			       (sleep-for 0 200)
+			       (goto-char (org-babel-where-is-src-block-result))
+			       (org-babel-read-result)))))))))
+
+(ert-deftest test-ob-python/async-named-output ()
+  (let (org-confirm-babel-evaluate
+        (org-babel-temporary-directory "/tmp")
+        (src-block "#+begin_src python :async :session :results output
+print(\"Yep!\")
+#+end_src")
+        (results-before "
+
+#+NAME: foobar
+#+RESULTS:
+: Nope!")
+        (results-after "
+
+#+NAME: foobar
+#+RESULTS:
+: Yep!
+"))
+    (org-test-with-temp-text
+     (concat src-block results-before)
+     (should (progn (org-babel-execute-src-block)
+                    (sleep-for 0 200)
+                    (string= (concat src-block results-after)
+                             (buffer-string)))))))
+
+(ert-deftest test-ob-python/async-output-drawer ()
+  (let (org-confirm-babel-evaluate
+        (org-babel-temporary-directory "/tmp")
+        (src-block "#+begin_src python :async :session :results output drawer
+print(list(range(3)))
+#+end_src")
+        (result "
+
+#+RESULTS:
+:results:
+[0, 1, 2]
+:end:
+"))
+    (org-test-with-temp-text
+     src-block
+     (should (progn (org-babel-execute-src-block)
+                    (sleep-for 0 200)
+                    (string= (concat src-block result)
+                             (buffer-string)))))))
+
 (provide 'test-ob-python)
 
 ;;; test-ob-python.el ends here
-- 
2.28.0

Re: [PATCH] Async session eval (2nd attempt)

[stardiviner]

This really is an good idea. Some other Babel languages like Ruby, JavaScript
might benefit from this ob-comint async evaluation. Awesome!

Jack Kamm <jackkamm@gmail.com> writes:

> This patch adds asynchronous evaluation for session blocks in
> Python. It also adds functionality to implement async session eval for
> other languages using ob-comint.el.
>
> To test the attached patch, add ":async" to a Python session block
> with a long computation (or "time.sleep") in it. Upon evaluation, your
> Emacs won't freeze to wait for the result -- instead, a placeholder
> will be inserted, and replaced with the true result when it's ready.
>
> I'll note how this is different from some related projects. ob-async
> implements asynchronous evaluation for Babel, but it doesn't work with
> sessions. emacs-jupyter, ein, and ob-ipython all implement
> asynchronous session evaluation, but only for Jupyter kernels. Jupyter
> is great for some cases, but sometimes I prefer to use the built-in
> org-babel languages without jupyter.
>
> The new functionality is mainly implemented in
> `org-babel-comint-async-filter', which I've defined in ob-comint.el,
> and added as a hook to `comint-output-filter-functions'.  Whenever new
> output is added to the comint buffer, the filter scans for an
> indicator token (this is inspired by
> `org-babel-comint-with-output'). Upon encountering the token, the
> filter uses a regular expression to extract a UUID or temp-file
> associated with the result, then searches for the appropriate location
> to add the result to.
>
> This is my 2nd attempt at this patch [0]. I have also ported it to an
> external package [1], but would like to have this functionality in Org
> proper, to permit better code reuse between async and sync
> implementations. The external package also includes an R
> implementation that I regularly use, as well as a Ruby implementation,
> but I've left these out to keep this initial patch smaller, and also I
> need to confirm copyright assignment on the Ruby implementation which
> was externally contributed.
>
> [0] https://orgmode.org/list/87muj04xim.fsf@jaheira.i-did-not-set--mail-host-address--so-tickle-me/
> [1] https://github.com/jackkamm/ob-session-async
>
> From 8b7695a148d1831c916737650e115833cb7fc752 Mon Sep 17 00:00:00 2001
> From: Jack Kamm <jackkamm@gmail.com>
> Date: Sun, 25 Oct 2020 11:40:10 -0700
> Subject: [PATCH] ob-comint.el, ob-python.el: Async session evaluation
>
> Adds functionality to ob-comint.el to implement async session eval on
> a per-language basis.  Adds a reference implementation for ob-python.
>
> * lisp/ob-comint.el (org-babel-comint-with-output): Remove comment.
> (org-babel-comint-async-indicator, org-babel-comint-async-buffers,
> org-babel-comint-async-file-callback,
> org-babel-comint-async-chunk-callback,
> org-babel-comint-async-dangling): Add buffer-local variables used for
> async comint evaluation.
> (org-babel-comint-use-async): Add function to determine whether block
> should be evaluated asynchronously.
> (org-babel-comint-async-filter): Add filter function to attach to
> comint-output-filter-functions for babel async eval.
> (org-babel-comint-async-register): Add function to setup buffer
> variables and hooks for session eval.
> (org-babel-comint-async-delete-dangling-and-eval): Add helper function
> for async session eval.
>
> * lisp/ob-python.el (org-babel-execute:python): Check for async header
> argument.
> (org-babel-python-evaluate): Check whether to use async evaluation.
> (org-babel-python-async-indicator): Add constant for indicating the
> start/end of async evaluations.
> (org-babel-python-async-evaluate-session): Add function for Python
> async eval.
>
> *
> testing/lisp/test-ob-python.el (test-ob-python/async-simple-session-output):
> Unit test for Python async session eval.
> (test-ob-python/async-named-output): Unit test that Python async eval
> can replace named output.
> (test-ob-python/async-output-drawer): Unit test that Python async eval
> works with drawer results.
> ---
>  etc/ORG-NEWS                   |  15 +++
>  lisp/ob-comint.el              | 172 +++++++++++++++++++++++++++++++--
>  lisp/ob-python.el              |  56 ++++++++++-
>  testing/lisp/test-ob-python.el |  61 ++++++++++++
>  4 files changed, 294 insertions(+), 10 deletions(-)
>
> diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
> index 7f935bf52..9d5fbbe30 100644
> --- a/etc/ORG-NEWS
> +++ b/etc/ORG-NEWS
> @@ -88,6 +88,21 @@ package, to convert pandas Dataframes into orgmode tables:
>  | 2 | 3 | 6 |
>  #+end_src
>  
> +*** Async session evaluation
> +
> +The =:async= header argument can be used for asynchronous evaluation
> +in session blocks for certain languages.
> +
> +Currently, async evaluation is supported in Python.  There is also
> +functionality to implement async evaluation in other languages that
> +use comint, but this needs to be done on a per-language basis.
> +
> +By default, async evaluation is disabled unless the =:async= header
> +argument is present.  You can also set =:async no= to force it off
> +(for example if you've set =:async= in a property drawer).
> +
> +Async evaluation is disabled during export.
> +
>  * Version 9.4
>  ** Incompatible changes
>  *** Possibly broken internal file links: please check and fix
> diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
> index d3484bb7c..591754dac 100644
> --- a/lisp/ob-comint.el
> +++ b/lisp/ob-comint.el
> @@ -94,12 +94,7 @@ (defmacro org-babel-comint-with-output (meta &rest body)
>  			       (regexp-quote ,eoe-indicator) nil t)
>  			      (re-search-forward
>  			       comint-prompt-regexp nil t)))))
> -	   (accept-process-output (get-buffer-process (current-buffer)))
> -	   ;; thought the following this would allow async
> -	   ;; background running, but I was wrong...
> -	   ;; (run-with-timer .5 .5 'accept-process-output
> -	   ;; 		 (get-buffer-process (current-buffer)))
> -	   )
> +	   (accept-process-output (get-buffer-process (current-buffer))))
>  	 ;; replace cut dangling text
>  	 (goto-char (process-mark (get-buffer-process (current-buffer))))
>  	 (insert dangling-text)
> @@ -149,6 +144,171 @@ (defun org-babel-comint-eval-invisibly-and-wait-for-file
>     (if (= (aref string (1- (length string))) ?\n) string (concat string "\n")))
>    (while (not (file-exists-p file)) (sit-for (or period 0.25))))
>  
> +
> +;; Async evaluation
> +
> +(defvar-local org-babel-comint-async-indicator nil
> +  "Regular expression that `org-babel-comint-async-filter' scans for.
> +It should have 2 parenthesized expressions,
> +e.g. \"org_babel_async_\\(start\\|end\\|file\\)_\\(.*\\)\". The
> +first parenthesized expression determines whether the token is
> +delimiting a result block, or whether the result is in a file. If
> +delimiting a block, the second expression gives a UUID for the
> +location to insert the result. Otherwise, the result is in a tmp
> +file, and the second expression gives the file name.")
> +
> +(defvar-local org-babel-comint-async-buffers nil
> +  "List of org-mode buffers to check for Babel async output results.")
> +
> +(defvar-local org-babel-comint-async-file-callback nil
> +  "Callback to clean and insert Babel async results from a temp file.
> +The callback function takes two arguments: the alist of params of the Babel
> +source block, and the name of the temp file.")
> +
> +(defvar-local org-babel-comint-async-chunk-callback nil
> +  "Callback function to clean Babel async output results before insertion.
> +Its single argument is a string consisting of output from the
> +comint process. It should return a string that will be be passed
> +to `org-babel-insert-result'.")
> +
> +(defvar-local org-babel-comint-async-dangling nil
> +  "Dangling piece of the last process output, in case
> +`org-babel-comint-async-indicator' is spread across multiple
> +comint outputs due to buffering.")
> +
> +(defun org-babel-comint-use-async (params)
> +  "Determine whether to use session async evaluation.
> +PARAMS are the header arguments as passed to
> +`org-babel-execute:lang'."
> +  (let ((async (assq :async params))
> +        (session (assq :session params)))
> +    (and async
> +	 (not org-babel-exp-reference-buffer)
> +         (not (equal (cdr async) "no"))
> +         (not (equal (cdr session) "none")))))
> +
> +(defun org-babel-comint-async-filter (string)
> +  "Captures Babel async output from comint buffer back to org-mode buffers.
> +This function is added as a hook to `comint-output-filter-functions'.
> +STRING contains the output originally inserted into the comint buffer."
> +  ;; Remove outdated org-mode buffers
> +  (setq org-babel-comint-async-buffers
> +	(cl-loop for buf in org-babel-comint-async-buffers
> +	      if (buffer-live-p buf)
> +	      collect buf))
> +  (let* ((indicator org-babel-comint-async-indicator)
> +	 (org-buffers org-babel-comint-async-buffers)
> +	 (file-callback org-babel-comint-async-file-callback)
> +	 (combined-string (concat org-babel-comint-async-dangling string))
> +	 (new-dangling combined-string)
> +	 ;; list of UUID's matched by `org-babel-comint-async-indicator'
> +	 uuid-list)
> +    (with-temp-buffer
> +      (insert combined-string)
> +      (goto-char (point-min))
> +      (while (re-search-forward indicator nil t)
> +	;; update dangling
> +	(setq new-dangling (buffer-substring (point) (point-max)))
> +	(cond ((equal (match-string 1) "end")
> +	       ;; save UUID for insertion later
> +	       (push (match-string 2) uuid-list))
> +	      ((equal (match-string 1) "file")
> +	       ;; insert results from tmp-file
> +	       (let ((tmp-file (match-string 2)))
> +		 (cl-loop for buf in org-buffers
> +		       until
> +		       (with-current-buffer buf
> +			 (save-excursion
> +			   (goto-char (point-min))
> +			   (when (search-forward tmp-file nil t)
> +			     (org-babel-previous-src-block)
> +                             (let* ((info (org-babel-get-src-block-info))
> +                                    (params (nth 2 info))
> +                                    (result-params
> +                                     (cdr (assq :result-params params))))
> +                               (org-babel-insert-result
> +                                 (funcall file-callback
> +                                          (nth
> +                                           2 (org-babel-get-src-block-info))
> +                                          tmp-file)
> +                                result-params info))
> +			     t))))))))
> +      ;; Truncate dangling to only the most recent output
> +      (when (> (length new-dangling) (length string))
> +	(setq new-dangling string)))
> +    (setq-local org-babel-comint-async-dangling new-dangling)
> +    (when uuid-list
> +      ;; Search for results in the comint buffer
> +      (save-excursion
> +	(goto-char (point-max))
> +	(while uuid-list
> +	  (re-search-backward indicator)
> +	  (when (equal (match-string 1) "end")
> +	    (let* ((uuid (match-string-no-properties 2))
> +		   (res-str-raw
> +		    (buffer-substring
> +		     ;; move point to beginning of indicator
> +                     (- (match-beginning 0) 1)
> +		     ;; find the matching start indicator
> +		     (cl-loop for pos = (re-search-backward indicator)
> +			   until (and (equal (match-string 1) "start")
> +				      (equal (match-string 2) uuid))
> +			   finally return (+ 1 (match-end 0)))))
> +		   ;; Apply callback to clean up the result
> +		   (res-str (funcall org-babel-comint-async-chunk-callback
> +                                     res-str-raw)))
> +	      ;; Search for uuid in associated org-buffers to insert results
> +	      (cl-loop for buf in org-buffers
> +		    until (with-current-buffer buf
> +			    (save-excursion
> +			      (goto-char (point-min))
> +			      (when (search-forward uuid nil t)
> +				(org-babel-previous-src-block)
> +                                (let* ((info (org-babel-get-src-block-info))
> +                                       (params (nth 2 info))
> +                                       (result-params
> +                                        (cdr (assq :result-params params))))
> +				  (org-babel-insert-result
> +                                   res-str result-params info))
> +				t))))
> +	      ;; Remove uuid from the list to search for
> +	      (setq uuid-list (delete uuid uuid-list)))))))))
> +
> +(defun org-babel-comint-async-register
> +    (session-buffer org-buffer indicator-regexp
> +		    chunk-callback file-callback)
> +  "Sets local org-babel-comint-async variables in SESSION-BUFFER.
> +ORG-BUFFER is added to `org-babel-comint-async-buffers' if not
> +present.  `org-babel-comint-async-indicator',
> +`org-babel-comint-async-chunk-callback', and
> +`org-babel-comint-async-file-callback' are set to
> +INDICATOR-REGEXP, CHUNK-CALLBACK, and FILE-CALLBACK
> +respectively."
> +  (org-babel-comint-in-buffer session-buffer
> +    (setq org-babel-comint-async-indicator indicator-regexp
> +	  org-babel-comint-async-chunk-callback chunk-callback
> +	  org-babel-comint-async-file-callback file-callback)
> +    (unless (memq org-buffer org-babel-comint-async-buffers)
> +      (setq org-babel-comint-async-buffers
> +	    (cons org-buffer org-babel-comint-async-buffers)))
> +    (add-hook 'comint-output-filter-functions
> +	      'org-babel-comint-async-filter nil t)))
> +
> +(defmacro org-babel-comint-async-delete-dangling-and-eval
> +    (session-buffer &rest body)
> +  "Remove dangling text in SESSION-BUFFER and evaluate BODY.
> +This is analogous to `org-babel-comint-with-output', but meant
> +for asynchronous output, and much shorter because inserting the
> +result is delegated to `org-babel-comint-async-filter'."
> +  (declare (indent 1))
> +  `(org-babel-comint-in-buffer ,session-buffer
> +     (goto-char (process-mark (get-buffer-process (current-buffer))))
> +     (delete-region (point) (point-max))
> +     ,@body))
> +(def-edebug-spec org-babel-comint-async-with-output (sexp body))
> +
>  (provide 'ob-comint)
>  
> +
> +
>  ;;; ob-comint.el ends here
> diff --git a/lisp/ob-python.el b/lisp/ob-python.el
> index 6752adc17..e26c34b64 100644
> --- a/lisp/ob-python.el
> +++ b/lisp/ob-python.el
> @@ -84,6 +84,7 @@ (defun org-babel-execute:python (body params)
>  	 (return-val (when (eq result-type 'value)
>  		       (cdr (assq :return params))))
>  	 (preamble (cdr (assq :preamble params)))
> +	 (async (org-babel-comint-use-async params))
>           (full-body
>  	  (concat
>  	   (org-babel-expand-body:generic
> @@ -92,7 +93,8 @@ (defun org-babel-execute:python (body params)
>  	   (when return-val
>  	     (format (if session "\n%s" "\nreturn %s") return-val))))
>           (result (org-babel-python-evaluate
> -		  session full-body result-type result-params preamble)))
> +		  session full-body result-type
> +		  result-params preamble async)))
>      (org-babel-reassemble-table
>       result
>       (org-babel-pick-name (cdr (assq :colname-names params))
> @@ -278,11 +280,14 @@ (defun org-babel-python-format-session-value
>  	  (if (member "pp" result-params) "True" "False")))
>  
>  (defun org-babel-python-evaluate
> -  (session body &optional result-type result-params preamble)
> +  (session body &optional result-type result-params preamble async)
>    "Evaluate BODY as Python code."
>    (if session
> -      (org-babel-python-evaluate-session
> -       session body result-type result-params)
> +      (if async
> +	  (org-babel-python-async-evaluate-session
> +	   session body result-type result-params)
> +	(org-babel-python-evaluate-session
> +	 session body result-type result-params))
>      (org-babel-python-evaluate-external-process
>       body result-type result-params preamble)))
>  
> @@ -391,6 +396,49 @@ (defun org-babel-python-read-string (string)
>        (substring string 1 -1)
>      string))
>  
> +;; Async session eval
> +
> +(defconst org-babel-python-async-indicator "print ('ob_comint_async_python_%s_%s')")
> +
> +(defun org-babel-python-async-value-callback (params tmp-file)
> +  (let ((result-params (cdr (assq :result-params params)))
> +	(results (org-babel-eval-read-file tmp-file)))
> +    (org-babel-result-cond result-params
> +      results
> +      (org-babel-python-table-or-string results))))
> +
> +(defun org-babel-python-async-evaluate-session
> +    (session body &optional result-type result-params)
> +  "Asynchronously evaluate BODY in SESSION.
> +Returns a placeholder string for insertion, to later be replaced
> +by `org-babel-comint-async-filter'."
> +  (org-babel-comint-async-register
> +   session (current-buffer)
> +   "ob_comint_async_python_\\(.+\\)_\\(.+\\)"
> +   'org-babel-chomp 'org-babel-python-async-value-callback)
> +  (let ((python-shell-buffer-name (org-babel-python-without-earmuffs session)))
> +    (pcase result-type
> +      (`output
> +       (let ((uuid (md5 (number-to-string (random 100000000)))))
> +         (with-temp-buffer
> +           (insert (format org-babel-python-async-indicator "start" uuid))
> +           (insert "\n")
> +           (insert body)
> +           (insert "\n")
> +           (insert (format org-babel-python-async-indicator "end" uuid))
> +           (python-shell-send-buffer))
> +         uuid))
> +      (`value
> +       (let ((tmp-results-file (org-babel-temp-file "python-"))
> +             (tmp-src-file (org-babel-temp-file "python-")))
> +         (with-temp-file tmp-src-file (insert body))
> +         (with-temp-buffer
> +           (insert (org-babel-python-format-session-value tmp-src-file tmp-results-file result-params))
> +           (insert "\n")
> +           (insert (format org-babel-python-async-indicator "file" tmp-results-file))
> +           (python-shell-send-buffer))
> +         tmp-results-file)))))
> +
>  (provide 'ob-python)
>  
>  ;;; ob-python.el ends here
> diff --git a/testing/lisp/test-ob-python.el b/testing/lisp/test-ob-python.el
> index a2cc7b79c..0267678cd 100644
> --- a/testing/lisp/test-ob-python.el
> +++ b/testing/lisp/test-ob-python.el
> @@ -207,6 +207,67 @@ (ert-deftest test-ob-python/session-value-sleep ()
>  #+end_src"
>  	    (org-babel-execute-src-block)))))
>  
> +(ert-deftest test-ob-python/async-simple-session-output ()
> +  (let ((org-babel-temporary-directory "/tmp")
> +        (org-confirm-babel-evaluate nil))
> +    (org-test-with-temp-text
> +     "#+begin_src python :session :async yes :results output
> +import time
> +time.sleep(.1)
> +print('Yep!')
> +#+end_src\n"
> +     (should (let ((expected "Yep!"))
> +	       (and (not (string= expected (org-babel-execute-src-block)))
> +		    (string= expected
> +			     (progn
> +			       (sleep-for 0 200)
> +			       (goto-char (org-babel-where-is-src-block-result))
> +			       (org-babel-read-result)))))))))
> +
> +(ert-deftest test-ob-python/async-named-output ()
> +  (let (org-confirm-babel-evaluate
> +        (org-babel-temporary-directory "/tmp")
> +        (src-block "#+begin_src python :async :session :results output
> +print(\"Yep!\")
> +#+end_src")
> +        (results-before "
> +
> +#+NAME: foobar
> +#+RESULTS:
> +: Nope!")
> +        (results-after "
> +
> +#+NAME: foobar
> +#+RESULTS:
> +: Yep!
> +"))
> +    (org-test-with-temp-text
> +     (concat src-block results-before)
> +     (should (progn (org-babel-execute-src-block)
> +                    (sleep-for 0 200)
> +                    (string= (concat src-block results-after)
> +                             (buffer-string)))))))
> +
> +(ert-deftest test-ob-python/async-output-drawer ()
> +  (let (org-confirm-babel-evaluate
> +        (org-babel-temporary-directory "/tmp")
> +        (src-block "#+begin_src python :async :session :results output drawer
> +print(list(range(3)))
> +#+end_src")
> +        (result "
> +
> +#+RESULTS:
> +:results:
> +[0, 1, 2]
> +:end:
> +"))
> +    (org-test-with-temp-text
> +     src-block
> +     (should (progn (org-babel-execute-src-block)
> +                    (sleep-for 0 200)
> +                    (string= (concat src-block result)
> +                             (buffer-string)))))))
> +
>  (provide 'test-ob-python)
>  
>  ;;; test-ob-python.el ends here


-- 
[ stardiviner ]
       I try to make every word tell the meaning that I want to express.

       Blog: https://stardiviner.github.io/
       IRC(freenode): stardiviner, Matrix: stardiviner
       GPG: F09F650D7D674819892591401B5DF1C95AE89AC3

Re: [PATCH] Async session eval (2nd attempt)

[Eric S Fraga]

On Monday, 26 Oct 2020 at 10:23, stardiviner wrote:
> This really is an good idea. Some other Babel languages like Ruby, JavaScript
> might benefit from this ob-comint async evaluation. Awesome!

Julia would benefit from this as well!  My Julia jobs usually take
minutes if not hours.  I usually tangle to run the code separately but
would love to simply C-c C-c and let it go.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-61-ga88806.dirty

Re: [PATCH] Async session eval (2nd attempt)

[Kyle Meyer]

Jack Kamm writes:

> This patch adds asynchronous evaluation for session blocks in
> Python. It also adds functionality to implement async session eval for
> other languages using ob-comint.el.
>
> To test the attached patch, add ":async" to a Python session block
> with a long computation (or "time.sleep") in it. Upon evaluation, your
> Emacs won't freeze to wait for the result -- instead, a placeholder
> will be inserted, and replaced with the true result when it's ready.
>
> [...]

Thanks.  Sounds exciting :)

So... have any Babel users on the list given this a spin?  Any comments
or feedback?

Re: [PATCH] Async session eval (2nd attempt)

[TEC]

Hi Jack,

I love the look of this! Thanks for submitting a patch.

Sorry it's taken so long for someone to take a look at it, I think a lot
of the 'main' Org people have been pretty busy over the last few months.

I just tried to give this a shot.
First up, I had to remove the ORG-NEWS part of the patch to be able to
provide it. It would be nice if you could update the patch so this
applies cleanly.

#+begin_example
error: patch failed: etc/ORG-NEWS:88
error: etc/ORG-NEWS: patch does not apply
#+end_example

To test this, after applying your patch (with ORG-NEWS removed), I
started emacs -Q, loaded Org, and opened a new file.

I was initially unable to get this to seem to work, until I changed the
:results type to "output".

See a excerpt from my test file below:

----- excerpt start -----

#+begin_src python :async :session blah :results output
from time import sleep

a=2

sleep(2)
print("Hi")
#+end_src

#+RESULTS:
: Hi

#+begin_src python :async :session blah
return(a)
#+end_src

#+RESULTS:
: /tmp/babel-62cQRX/python-EfJ4o4

#+begin_src python :async :session blah :results output
print(a)
#+end_src

#+RESULTS:
: 2

----- excerpt end -----


I'm surprised this didn't work with the non-output block though.

Other than this, I'm rather happy to see that when I tried to execute
two long running blocks at once, the second one was not executed until
the first completed :)

Finally, I see that this requires :session to be set in order to work.
Might it be possible to have this work for non-session blocks too? It
seems odd that what I'd imagine is the harder case (session blocks) is
supported, but one-shot (non-session) blocks aren't.

Thanks again for your work, and I look forward to seeing what else you
have in the future!

--
Timothy

p.s. After this is merged, it would be great to see support for other
languages grow :)

Jack Kamm <jackkamm@gmail.com> writes:

> This patch adds asynchronous evaluation for session blocks in
> Python. It also adds functionality to implement async session eval for
> other languages using ob-comint.el.
>
> To test the attached patch, add ":async" to a Python session block
> with a long computation (or "time.sleep") in it. Upon evaluation, your
> Emacs won't freeze to wait for the result -- instead, a placeholder
> will be inserted, and replaced with the true result when it's ready.
>
> I'll note how this is different from some related projects. ob-async
> implements asynchronous evaluation for Babel, but it doesn't work with
> sessions. emacs-jupyter, ein, and ob-ipython all implement
> asynchronous session evaluation, but only for Jupyter kernels. Jupyter
> is great for some cases, but sometimes I prefer to use the built-in
> org-babel languages without jupyter.
>
> The new functionality is mainly implemented in
> `org-babel-comint-async-filter', which I've defined in ob-comint.el,
> and added as a hook to `comint-output-filter-functions'.  Whenever new
> output is added to the comint buffer, the filter scans for an
> indicator token (this is inspired by
> `org-babel-comint-with-output'). Upon encountering the token, the
> filter uses a regular expression to extract a UUID or temp-file
> associated with the result, then searches for the appropriate location
> to add the result to.
>
> This is my 2nd attempt at this patch [0]. I have also ported it to an
> external package [1], but would like to have this functionality in Org
> proper, to permit better code reuse between async and sync
> implementations. The external package also includes an R
> implementation that I regularly use, as well as a Ruby implementation,
> but I've left these out to keep this initial patch smaller, and also I
> need to confirm copyright assignment on the Ruby implementation which
> was externally contributed.
>
> [0] https://orgmode.org/list/87muj04xim.fsf@jaheira.i-did-not-set--mail-host-address--so-tickle-me/
> [1] https://github.com/jackkamm/ob-session-async

Re: [PATCH] Async session eval (2nd attempt)

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 1712 bytes --]

Hi Timothy,

Many thanks for testing this out.

> I just tried to give this a shot.
> First up, I had to remove the ORG-NEWS part of the patch to be able to
> provide it. It would be nice if you could update the patch so this
> applies cleanly.

I'm attaching an updated patch rebased on master.

> I was initially unable to get this to seem to work, until I changed the
> :results type to "output".
>
> #+begin_src python :async :session blah
> return(a)
> #+end_src
>
> #+RESULTS:
> : /tmp/babel-62cQRX/python-EfJ4o4

ob-python session blocks don't use "return", so this should just be:

#+begin_src python :async :session blah
a
#+end_src

#+RESULTS:
: 2

> Finally, I see that this requires :session to be set in order to work.
> Might it be possible to have this work for non-session blocks too? It
> seems odd that what I'd imagine is the harder case (session blocks) is
> supported, but one-shot (non-session) blocks aren't.

The non-session case is substantially different, and I think it would
probably require a separate implementation.

One possible approach would be to modify ob-eval.el, so that
org-babel--shell-command-on-region uses make-process instead of
process-file.

I agree it would be nice to have, but it would take a bit of work to
figure it all out, and there is already ob-async.el [1] that implements
non-session async for all languages. (I wish it could be brought into
org-mode, but it probably can't, because it depends on the external
async.el.)

> p.s. After this is merged, it would be great to see support for other
> languages grow :)

I also have an async implementation for ob-R that's ready after this is
merged :)

Cheers,
Jack

[1] https://github.com/astahlman/ob-async


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-comint.el-ob-python.el-Async-session-evaluation.patch --]
[-- Type: text/x-patch, Size: 17230 bytes --]

From 864a2377b4eea58df6b0ccd07c4bcba080ecc724 Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Sun, 28 Feb 2021 13:17:33 -0800
Subject: [PATCH] ob-comint.el, ob-python.el: Async session evaluation

Adds functionality to ob-comint.el to implement async session eval on
a per-language basis.  Adds a reference implementation for ob-python.

* lisp/ob-comint.el (org-babel-comint-with-output): Remove comment.
(org-babel-comint-async-indicator, org-babel-comint-async-buffers,
org-babel-comint-async-file-callback,
org-babel-comint-async-chunk-callback,
org-babel-comint-async-dangling): Add buffer-local variables used for
async comint evaluation.
(org-babel-comint-use-async): Add function to determine whether block
should be evaluated asynchronously.
(org-babel-comint-async-filter): Add filter function to attach to
comint-output-filter-functions for babel async eval.
(org-babel-comint-async-register): Add function to setup buffer
variables and hooks for session eval.
(org-babel-comint-async-delete-dangling-and-eval): Add helper function
for async session eval.

* lisp/ob-python.el (org-babel-execute:python): Check for async header
argument.
(org-babel-python-evaluate): Check whether to use async evaluation.
(org-babel-python-async-indicator): Add constant for indicating the
start/end of async evaluations.
(org-babel-python-async-evaluate-session): Add function for Python
async eval.

*
testing/lisp/test-ob-python.el (test-ob-python/async-simple-session-output):
Unit test for Python async session eval.
(test-ob-python/async-named-output): Unit test that Python async eval
can replace named output.
(test-ob-python/async-output-drawer): Unit test that Python async eval
works with drawer results.
---
 etc/ORG-NEWS                   |  15 +++
 lisp/ob-comint.el              | 173 +++++++++++++++++++++++++++++++--
 lisp/ob-python.el              |  56 ++++++++++-
 testing/lisp/test-ob-python.el |  61 ++++++++++++
 4 files changed, 295 insertions(+), 10 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index f95a568a6..eff75605c 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -169,6 +169,21 @@ tags including from both buffer local and user defined persistent
 global list (~org-tag-alist~ and ~org-tag-persistent-alist~).  Now
 option ~org-complete-tags-always-offer-all-agenda-tags~ is honored.
 
+*** Async session evaluation
+
+The =:async= header argument can be used for asynchronous evaluation
+in session blocks for certain languages.
+
+Currently, async evaluation is supported in Python.  There is also
+functionality to implement async evaluation in other languages that
+use comint, but this needs to be done on a per-language basis.
+
+By default, async evaluation is disabled unless the =:async= header
+argument is present.  You can also set =:async no= to force it off
+(for example if you've set =:async= in a property drawer).
+
+Async evaluation is disabled during export.
+
 ** Miscellaneous
 *** =org-goto-first-child= now works before first heading
 
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index b14849df6..d81ff3edd 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -93,12 +93,7 @@ (defmacro org-babel-comint-with-output (meta &rest body)
 			       (regexp-quote ,eoe-indicator) nil t)
 			      (re-search-forward
 			       comint-prompt-regexp nil t)))))
-	   (accept-process-output (get-buffer-process (current-buffer)))
-	   ;; thought the following this would allow async
-	   ;; background running, but I was wrong...
-	   ;; (run-with-timer .5 .5 'accept-process-output
-	   ;; 		 (get-buffer-process (current-buffer)))
-	   )
+	   (accept-process-output (get-buffer-process (current-buffer))))
 	 ;; replace cut dangling text
 	 (goto-char (process-mark (get-buffer-process (current-buffer))))
 	 (insert dangling-text)
@@ -147,6 +142,172 @@ (defun org-babel-comint-eval-invisibly-and-wait-for-file
    (if (= (aref string (1- (length string))) ?\n) string (concat string "\n")))
   (while (not (file-exists-p file)) (sit-for (or period 0.25))))
 
+
+;; Async evaluation
+
+(defvar-local org-babel-comint-async-indicator nil
+  "Regular expression that `org-babel-comint-async-filter' scans for.
+It should have 2 parenthesized expressions,
+e.g. \"org_babel_async_\\(start\\|end\\|file\\)_\\(.*\\)\". The
+first parenthesized expression determines whether the token is
+delimiting a result block, or whether the result is in a file. If
+delimiting a block, the second expression gives a UUID for the
+location to insert the result. Otherwise, the result is in a tmp
+file, and the second expression gives the file name.")
+
+(defvar-local org-babel-comint-async-buffers nil
+  "List of org-mode buffers to check for Babel async output results.")
+
+(defvar-local org-babel-comint-async-file-callback nil
+  "Callback to clean and insert Babel async results from a temp file.
+The callback function takes two arguments: the alist of params of the Babel
+source block, and the name of the temp file.")
+
+(defvar-local org-babel-comint-async-chunk-callback nil
+  "Callback function to clean Babel async output results before insertion.
+Its single argument is a string consisting of output from the
+comint process. It should return a string that will be be passed
+to `org-babel-insert-result'.")
+
+(defvar-local org-babel-comint-async-dangling nil
+  "Dangling piece of the last process output, in case
+`org-babel-comint-async-indicator' is spread across multiple
+comint outputs due to buffering.")
+
+(defun org-babel-comint-use-async (params)
+  "Determine whether to use session async evaluation.
+PARAMS are the header arguments as passed to
+`org-babel-execute:lang'."
+  (let ((async (assq :async params))
+        (session (assq :session params)))
+    (and async
+	 (not org-babel-exp-reference-buffer)
+         (not (equal (cdr async) "no"))
+         (not (equal (cdr session) "none")))))
+
+(defun org-babel-comint-async-filter (string)
+  "Captures Babel async output from comint buffer back to org-mode buffers.
+This function is added as a hook to `comint-output-filter-functions'.
+STRING contains the output originally inserted into the comint buffer."
+  ;; Remove outdated org-mode buffers
+  (setq org-babel-comint-async-buffers
+	(cl-loop for buf in org-babel-comint-async-buffers
+	      if (buffer-live-p buf)
+	      collect buf))
+  (let* ((indicator org-babel-comint-async-indicator)
+	 (org-buffers org-babel-comint-async-buffers)
+	 (file-callback org-babel-comint-async-file-callback)
+	 (combined-string (concat org-babel-comint-async-dangling string))
+	 (new-dangling combined-string)
+	 ;; list of UUID's matched by `org-babel-comint-async-indicator'
+	 uuid-list)
+    (with-temp-buffer
+      (insert combined-string)
+      (goto-char (point-min))
+      (while (re-search-forward indicator nil t)
+	;; update dangling
+	(setq new-dangling (buffer-substring (point) (point-max)))
+	(cond ((equal (match-string 1) "end")
+	       ;; save UUID for insertion later
+	       (push (match-string 2) uuid-list))
+	      ((equal (match-string 1) "file")
+	       ;; insert results from tmp-file
+	       (let ((tmp-file (match-string 2)))
+		 (cl-loop for buf in org-buffers
+		       until
+		       (with-current-buffer buf
+			 (save-excursion
+			   (goto-char (point-min))
+			   (when (search-forward tmp-file nil t)
+			     (org-babel-previous-src-block)
+                             (let* ((info (org-babel-get-src-block-info))
+                                    (params (nth 2 info))
+                                    (result-params
+                                     (cdr (assq :result-params params))))
+                               (org-babel-insert-result
+                                 (funcall file-callback
+                                          (nth
+                                           2 (org-babel-get-src-block-info))
+                                          tmp-file)
+                                result-params info))
+			     t))))))))
+      ;; Truncate dangling to only the most recent output
+      (when (> (length new-dangling) (length string))
+	(setq new-dangling string)))
+    (setq-local org-babel-comint-async-dangling new-dangling)
+    (when uuid-list
+      ;; Search for results in the comint buffer
+      (save-excursion
+	(goto-char (point-max))
+	(while uuid-list
+	  (re-search-backward indicator)
+	  (when (equal (match-string 1) "end")
+	    (let* ((uuid (match-string-no-properties 2))
+		   (res-str-raw
+		    (buffer-substring
+		     ;; move point to beginning of indicator
+                     (- (match-beginning 0) 1)
+		     ;; find the matching start indicator
+		     (cl-loop
+                      do (re-search-backward indicator)
+		      until (and (equal (match-string 1) "start")
+				 (equal (match-string 2) uuid))
+		      finally return (+ 1 (match-end 0)))))
+		   ;; Apply callback to clean up the result
+		   (res-str (funcall org-babel-comint-async-chunk-callback
+                                     res-str-raw)))
+	      ;; Search for uuid in associated org-buffers to insert results
+	      (cl-loop for buf in org-buffers
+		    until (with-current-buffer buf
+			    (save-excursion
+			      (goto-char (point-min))
+			      (when (search-forward uuid nil t)
+				(org-babel-previous-src-block)
+                                (let* ((info (org-babel-get-src-block-info))
+                                       (params (nth 2 info))
+                                       (result-params
+                                        (cdr (assq :result-params params))))
+				  (org-babel-insert-result
+                                   res-str result-params info))
+				t))))
+	      ;; Remove uuid from the list to search for
+	      (setq uuid-list (delete uuid uuid-list)))))))))
+
+(defun org-babel-comint-async-register
+    (session-buffer org-buffer indicator-regexp
+		    chunk-callback file-callback)
+  "Sets local org-babel-comint-async variables in SESSION-BUFFER.
+ORG-BUFFER is added to `org-babel-comint-async-buffers' if not
+present.  `org-babel-comint-async-indicator',
+`org-babel-comint-async-chunk-callback', and
+`org-babel-comint-async-file-callback' are set to
+INDICATOR-REGEXP, CHUNK-CALLBACK, and FILE-CALLBACK
+respectively."
+  (org-babel-comint-in-buffer session-buffer
+    (setq org-babel-comint-async-indicator indicator-regexp
+	  org-babel-comint-async-chunk-callback chunk-callback
+	  org-babel-comint-async-file-callback file-callback)
+    (unless (memq org-buffer org-babel-comint-async-buffers)
+      (setq org-babel-comint-async-buffers
+	    (cons org-buffer org-babel-comint-async-buffers)))
+    (add-hook 'comint-output-filter-functions
+	      'org-babel-comint-async-filter nil t)))
+
+(defmacro org-babel-comint-async-delete-dangling-and-eval
+    (session-buffer &rest body)
+  "Remove dangling text in SESSION-BUFFER and evaluate BODY.
+This is analogous to `org-babel-comint-with-output', but meant
+for asynchronous output, and much shorter because inserting the
+result is delegated to `org-babel-comint-async-filter'."
+  (declare (indent 1))
+  `(org-babel-comint-in-buffer ,session-buffer
+     (goto-char (process-mark (get-buffer-process (current-buffer))))
+     (delete-region (point) (point-max))
+     ,@body))
+(def-edebug-spec org-babel-comint-async-with-output (sexp body))
+
 (provide 'ob-comint)
 
+
+
 ;;; ob-comint.el ends here
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 9f2386392..45c33a453 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -84,6 +84,7 @@ (defun org-babel-execute:python (body params)
 	 (return-val (when (eq result-type 'value)
 		       (cdr (assq :return params))))
 	 (preamble (cdr (assq :preamble params)))
+	 (async (org-babel-comint-use-async params))
          (full-body
 	  (concat
 	   (org-babel-expand-body:generic
@@ -92,7 +93,8 @@ (defun org-babel-execute:python (body params)
 	   (when return-val
 	     (format (if session "\n%s" "\nreturn %s") return-val))))
          (result (org-babel-python-evaluate
-		  session full-body result-type result-params preamble)))
+		  session full-body result-type
+		  result-params preamble async)))
     (org-babel-reassemble-table
      result
      (org-babel-pick-name (cdr (assq :colname-names params))
@@ -278,11 +280,14 @@ (defun org-babel-python-format-session-value
 	  (if (member "pp" result-params) "True" "False")))
 
 (defun org-babel-python-evaluate
-  (session body &optional result-type result-params preamble)
+  (session body &optional result-type result-params preamble async)
   "Evaluate BODY as Python code."
   (if session
-      (org-babel-python-evaluate-session
-       session body result-type result-params)
+      (if async
+	  (org-babel-python-async-evaluate-session
+	   session body result-type result-params)
+	(org-babel-python-evaluate-session
+	 session body result-type result-params))
     (org-babel-python-evaluate-external-process
      body result-type result-params preamble)))
 
@@ -391,6 +396,49 @@ (defun org-babel-python-read-string (string)
       (substring string 1 -1)
     string))
 
+;; Async session eval
+
+(defconst org-babel-python-async-indicator "print ('ob_comint_async_python_%s_%s')")
+
+(defun org-babel-python-async-value-callback (params tmp-file)
+  (let ((result-params (cdr (assq :result-params params)))
+	(results (org-babel-eval-read-file tmp-file)))
+    (org-babel-result-cond result-params
+      results
+      (org-babel-python-table-or-string results))))
+
+(defun org-babel-python-async-evaluate-session
+    (session body &optional result-type result-params)
+  "Asynchronously evaluate BODY in SESSION.
+Returns a placeholder string for insertion, to later be replaced
+by `org-babel-comint-async-filter'."
+  (org-babel-comint-async-register
+   session (current-buffer)
+   "ob_comint_async_python_\\(.+\\)_\\(.+\\)"
+   'org-babel-chomp 'org-babel-python-async-value-callback)
+  (let ((python-shell-buffer-name (org-babel-python-without-earmuffs session)))
+    (pcase result-type
+      (`output
+       (let ((uuid (md5 (number-to-string (random 100000000)))))
+         (with-temp-buffer
+           (insert (format org-babel-python-async-indicator "start" uuid))
+           (insert "\n")
+           (insert body)
+           (insert "\n")
+           (insert (format org-babel-python-async-indicator "end" uuid))
+           (python-shell-send-buffer))
+         uuid))
+      (`value
+       (let ((tmp-results-file (org-babel-temp-file "python-"))
+             (tmp-src-file (org-babel-temp-file "python-")))
+         (with-temp-file tmp-src-file (insert body))
+         (with-temp-buffer
+           (insert (org-babel-python-format-session-value tmp-src-file tmp-results-file result-params))
+           (insert "\n")
+           (insert (format org-babel-python-async-indicator "file" tmp-results-file))
+           (python-shell-send-buffer))
+         tmp-results-file)))))
+
 (provide 'ob-python)
 
 ;;; ob-python.el ends here
diff --git a/testing/lisp/test-ob-python.el b/testing/lisp/test-ob-python.el
index a2cc7b79c..0267678cd 100644
--- a/testing/lisp/test-ob-python.el
+++ b/testing/lisp/test-ob-python.el
@@ -207,6 +207,67 @@ (ert-deftest test-ob-python/session-value-sleep ()
 #+end_src"
 	    (org-babel-execute-src-block)))))
 
+(ert-deftest test-ob-python/async-simple-session-output ()
+  (let ((org-babel-temporary-directory "/tmp")
+        (org-confirm-babel-evaluate nil))
+    (org-test-with-temp-text
+     "#+begin_src python :session :async yes :results output
+import time
+time.sleep(.1)
+print('Yep!')
+#+end_src\n"
+     (should (let ((expected "Yep!"))
+	       (and (not (string= expected (org-babel-execute-src-block)))
+		    (string= expected
+			     (progn
+			       (sleep-for 0 200)
+			       (goto-char (org-babel-where-is-src-block-result))
+			       (org-babel-read-result)))))))))
+
+(ert-deftest test-ob-python/async-named-output ()
+  (let (org-confirm-babel-evaluate
+        (org-babel-temporary-directory "/tmp")
+        (src-block "#+begin_src python :async :session :results output
+print(\"Yep!\")
+#+end_src")
+        (results-before "
+
+#+NAME: foobar
+#+RESULTS:
+: Nope!")
+        (results-after "
+
+#+NAME: foobar
+#+RESULTS:
+: Yep!
+"))
+    (org-test-with-temp-text
+     (concat src-block results-before)
+     (should (progn (org-babel-execute-src-block)
+                    (sleep-for 0 200)
+                    (string= (concat src-block results-after)
+                             (buffer-string)))))))
+
+(ert-deftest test-ob-python/async-output-drawer ()
+  (let (org-confirm-babel-evaluate
+        (org-babel-temporary-directory "/tmp")
+        (src-block "#+begin_src python :async :session :results output drawer
+print(list(range(3)))
+#+end_src")
+        (result "
+
+#+RESULTS:
+:results:
+[0, 1, 2]
+:end:
+"))
+    (org-test-with-temp-text
+     src-block
+     (should (progn (org-babel-execute-src-block)
+                    (sleep-for 0 200)
+                    (string= (concat src-block result)
+                             (buffer-string)))))))
+
 (provide 'test-ob-python)
 
 ;;; test-ob-python.el ends here
-- 
2.30.1

Re: [PATCH] Async session eval (2nd attempt)

[Timothy]

Jack Kamm <jackkamm@gmail.com> writes:

> I'm attaching an updated patch rebased on master.

Fantastic. Thanks for doing this.

> ob-python session blocks don't use "return", so this should just be:

Ah, oops 😛

> The non-session case is substantially different, and I think it would
> probably require a separate implementation.

I see, fair enough. I haven't looked at the internals myself.

> One possible approach would be to modify ob-eval.el, so that
> org-babel--shell-command-on-region uses make-process instead of
> process-file.
>
> I agree it would be nice to have, but it would take a bit of work to
> figure it all out, and there is already ob-async.el [1] that implements
> non-session async for all languages. (I wish it could be brought into
> org-mode, but it probably can't, because it depends on the external
> async.el.)

Interesting that you use a different async method.

>> p.s. After this is merged, it would be great to see support for other
>> languages grow :)
>
> I also have an async implementation for ob-R that's ready after this is
> merged :)

Brilliant! I can't wait.

I know Bastien has found himself unfortunately busy as of late (which I
think is why he's looking to have someone else take over as project
lead), but please don't be disheartened by the slow progress with
getting this merged! I can assure you that quite a few people are
watching this with keen interest :)

All the best,

Timothy.

Re: [PATCH] Async session eval (2nd attempt)

[Kyle Meyer]

Jack Kamm writes:

> I also have an async implementation for ob-R that's ready after this is
> merged :)

It's a bit disappointing that only one babel user has tested this out
and provided feedback, but please feel free to merge this whenever you
think it's ready.

I'm very happy to leave the babel details to you, but here are minor
comments from a quick read-through looking for things that will likely
be changed/cleaned up (either in the Org repo or the Emacs repo) if left
as is.

> Subject: [PATCH] ob-comint.el, ob-python.el: Async session evaluation
[...]
> +;; Async evaluation

For a heading comment, please use at least three semicolons.

  (info "(elisp)Comment Tips")

> +
> +(defvar-local org-babel-comint-async-indicator nil
> +  "Regular expression that `org-babel-comint-async-filter' scans for.
> +It should have 2 parenthesized expressions,
> +e.g. \"org_babel_async_\\(start\\|end\\|file\\)_\\(.*\\)\". The
> +first parenthesized expression determines whether the token is
> +delimiting a result block, or whether the result is in a file. If
> +delimiting a block, the second expression gives a UUID for the
> +location to insert the result. Otherwise, the result is in a tmp
> +file, and the second expression gives the file name.")
> +
> +(defvar-local org-babel-comint-async-buffers nil
> +  "List of org-mode buffers to check for Babel async output results.")

s/org-mode/Org mode/ here and other spots, following
doc/Documentation_Standards.org and tree-wide cleanups like de24694f0
(Turn org-mode into Org or Org mode, 2016-08-23).

Also, you're missing two spaces between some sentences.

> +(defmacro org-babel-comint-async-delete-dangling-and-eval
> +    (session-buffer &rest body)
> +  "Remove dangling text in SESSION-BUFFER and evaluate BODY.
> +This is analogous to `org-babel-comint-with-output', but meant
> +for asynchronous output, and much shorter because inserting the
> +result is delegated to `org-babel-comint-async-filter'."
> +  (declare (indent 1))
> +  `(org-babel-comint-in-buffer ,session-buffer
> +     (goto-char (process-mark (get-buffer-process (current-buffer))))
> +     (delete-region (point) (point-max))
> +     ,@body))
> +(def-edebug-spec org-babel-comint-async-with-output (sexp body))

Please move this edebug spec to the `declare' form (see 7dd1cfb6c,
2021-02-12).

> diff --git a/testing/lisp/test-ob-python.el b/testing/lisp/test-ob-python.el
> index a2cc7b79c..0267678cd 100644
> --- a/testing/lisp/test-ob-python.el
> +++ b/testing/lisp/test-ob-python.el
> @@ -207,6 +207,67 @@ (ert-deftest test-ob-python/session-value-sleep ()
>  #+end_src"
>  	    (org-babel-execute-src-block)))))
>  
> +(ert-deftest test-ob-python/async-simple-session-output ()
> +  (let ((org-babel-temporary-directory "/tmp")

Prefer `temporary-file-directory' to hard coding "/tmp".

Thanks.