It seems, something goes a bit wrong. However it is rather confusion than anything really broken. Maybe everybody uses helm, ivy, etc., so nobody is affected. Usually I jump from one note to another heading with C-u C-c C-j (org-goto interface with target completion). I have not setup helm or other similar package. Completion works, maybe it is not really perfect (e.g. completion of top-level headings and deeper ones differ a bit). I have heard that C-u C-c C-w (org-refile) could work in a similar way. I did not like the latter variant due to mandatory file name in the beginning of a target. I have tried to enable org-refile-use-cache. Now completion options depends on what method was called earlier: org-refile or org-goto. If after cache clean up (C-u C-u C-u C-c C-w) first command was C-u C-c C-j (org-goto) than there is no file name as a prefix for both commands. However file name is prepended for both C-u C-c C-j and C-u C-c C-w if at first org-refile was called. There is one issue however. Default option option does not work if after cache clean other command is called, e.g. - jump using C-u C-c C-j - clean cache C-u C-u C-u C-c C-w - try to jump or to refile [C-u] C-c C-w to default offered option - "user-error: Invalid target location" I have just one file in the org-agenda-files list. It is becoming larger, that is why I decided to try org-refile-use-cache. My customization is minimal (besides org-default-dir and org-agenda-files): (org-refile-targets (quote ((org-agenda-files :maxlevel . 5)))) (org-outline-path-complete-in-steps nil) I have tried to add (nil :maxlevel . 5) to org-refile-targets to check if options without file name will appear in addition to ones with file name for org-refile, but it has not happened. It was not clear at first that it is possible to start query with "/" and file name is added when TAB is pressed. Is it expected that with cache enabled, command that causes cache update affects appearance of completion options to its sibling (org-refile and org-goto)?
idk if thi will help buit a fwe not3es. just in case they are relevant to your case. fwiw i have not heard of interactions between org-goto and the refile mechanism and the cache. i hafve not heard that the latter used a cache but i do not use it. refile with and withoyut the goto prefix should normatively be congruent and both use cache. i used to use the refile cache, then discovered that ido-hacks sped ido so much that i didn't need it. it would cache one fileset and then be potentially not usable for another. sort of like one cache and multiple callers. idk if htat got fixed. probably not used much. until recently in maint, ido and ido hacks with both refile and refile goto [note: org-refile with a goto arg, not org-goto] has worked perfectly. with no cache. now, there is an issue, where with no cache that i know of, the first use, or the first use in a long time, will actually present a huge file list that includes crazy elements and is not constrained by even the verify function. thus if i type faster than it goes, the wrong item gets selected. even if i use the same selectors as normal. so there's some bug or race condition. it is slow at those times. this might not be relevant to your case but seems worth mentioning. i think i have not filed a bug report as i don't know how to repro it at this time. On 3/2/21, Maxim Nikulin <manikulin@gmail.com> wrote: > It seems, something goes a bit wrong. However it is rather confusion > than anything really broken. > > Maybe everybody uses helm, ivy, etc., so nobody is affected. > > Usually I jump from one note to another heading with C-u C-c C-j > (org-goto interface with target completion). I have not setup helm or > other similar package. Completion works, maybe it is not really perfect > (e.g. completion of top-level headings and deeper ones differ a bit). I > have heard that C-u C-c C-w (org-refile) could work in a similar way. I > did not like the latter variant due to mandatory file name in the > beginning of a target. > > I have tried to enable org-refile-use-cache. Now completion options > depends on what method was called earlier: org-refile or org-goto. If > after cache clean up (C-u C-u C-u C-c C-w) first command was C-u C-c C-j > (org-goto) than there is no file name as a prefix for both commands. > However file name is prepended for both C-u C-c C-j and C-u C-c C-w if > at first org-refile was called. > > There is one issue however. Default option option does not work if after > cache clean other command is called, e.g. > - jump using C-u C-c C-j > - clean cache C-u C-u C-u C-c C-w > - try to jump or to refile [C-u] C-c C-w to default offered option > - "user-error: Invalid target location" > > I have just one file in the org-agenda-files list. It is becoming > larger, that is why I decided to try org-refile-use-cache. My > customization is minimal (besides org-default-dir and org-agenda-files): > > (org-refile-targets (quote ((org-agenda-files :maxlevel . 5)))) > (org-outline-path-complete-in-steps nil) > > I have tried to add (nil :maxlevel . 5) to org-refile-targets to check > if options without file name will appear in addition to ones with file > name for org-refile, but it has not happened. It was not clear at first > that it is possible to start query with "/" and file name is added when > TAB is pressed. > > Is it expected that with cache enabled, command that causes cache update > affects appearance of completion options to its sibling (org-refile and > org-goto)? > > > -- The Kafka Pandemic Please learn what misopathy is. https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html
On 03/03/2021 09:34, Samuel Wales wrote:
>
> until recently in maint, ido and ido hacks with both refile and refile
> goto [note: org-refile with a goto arg, not org-goto] has worked
> perfectly. with no cache. now, there is an issue, where with no
> cache that i know of, the first use, or the first use in a long time,
> will actually present a huge file list that includes crazy elements
> and is not constrained by even the verify function.
Concerning performance. After reading your message I have realized that
it should be really fast to extract several thousand headings from a
buffer using regexp. However actually it is not so:
(benchmark-run 10 (and (org-refile-get-targets) nil))
| 9.868742533999999 | 40 | 2.999932755999996 |
So preparing the list of refile targets currently takes almost precisely
1 second. It is slow. Results are same for org-9.1.6 and
release_9.4.4-231-gf46925. I am surprised however that 9.1.6 and 9.3.1
are installed as system packages and have compiled files. I have not
compiled git version but it works with the same speed.
(length (org-refile-get-targets))
: 3220
Preferences:
(require 'org-refile)
(setq org-agenda-files '("~/notes/notes.org"))
(setq org-refile-use-cache nil)
(setq org-refile-use-outline-path t)
(setq org-outline-path-complete-in-steps nil)
(setq org-refile-targets '((org-agenda-files :maxlevel . 5)))
Some lines from profile:
- org-refile-get-targets 8414 74%
... - org-get-outline-path 7403 65%
... - org--get-outline-path-1 7286 64%
... - org-up-heading-safe 6328 56%
There is a room for improvement. Outline paths could be obtained in a
single pass without backward search. It should speed up building the
list of targets by 2 or 3 times.
Though it is unrelated to issues with default option and cache entries
when both ways to call org-refile are used: directly and through org-goto.
On 03/03/2021 00:15, Maxim Nikulin wrote:
>
> There is one issue however. Default option option does not work if after
> cache clean other command is called, e.g.
> - jump using C-u C-c C-j
> - clean cache C-u C-u C-u C-c C-w
> - try to jump or to refile [C-u] C-c C-w to default offered option
> - "user-error: Invalid target location"
I have realized that this issue is rather loosely related to
org-refile-use-cache and I have seen it in a worse variant with disabled
cache.
file init-refile.el:
(package-initialize)
(custom-set-variables
'(org-agenda-files (quote ("~/examples/org/test-notes.org")))
'(org-capture-templates
(quote
(("t" "Test" entry
(file "")
"* %?\n\n%U\n" :empty-lines 1))))
'(org-default-notes-file "~/examples/org/capture.org")
'(org-directory "~/examples/org/")
'(org-modules (quote (org-refile)))
'(org-outline-path-complete-in-steps nil)
'(org-refile-targets (quote ((nil :maxlevel . 5) (org-agenda-files
:maxlevel . 5))))
'(org-refile-use-outline-path t)
)
emacs -Q -L ~/src/emacs/org-mode/lisp/ -L
~/src/emacs/org-mode/contrib/lisp/ -l ~/examples/org/init-refile.el
~/examples/org/test-notes.org
- C-u C-c C-j (org-goto) to some heading in the test-notes.org, e.g.
"Two" / "Third" / "Theme"
- M-x org-capture RET t to capture some note
- Let's assume that target of previously executed org-goto suits for
this note
- C-c C-w RET to refile the note from capture.org file to the heading in
test-notes.org offered as the default option.
Actual result:
- "Invalid target location"
- Captured note is still in the capture.org file but it is not apparent
since capture frame is closed.
Expected result:
- Default option works for refile even if it is remained from org-goto
command
Side note. In some cases it not so easy to close capture frame. If
org-default-note-file is configured to a file in a non-existing
directory then C-c C-k asks to create that directory but does not
discards the note.
interesting. that would be great to speed it up. [i just meant that the file list used to be correct.] but it is slow these days. for me, (benchmark-run 1 (length (org-refile-get-targets))) is 8s for 5886 targets. i presume you mean something like that up heading is unnecessary because you can keep a running olpath as you search. On 3/4/21, Maxim Nikulin <manikulin@gmail.com> wrote: > On 03/03/2021 09:34, Samuel Wales wrote: >> >> until recently in maint, ido and ido hacks with both refile and refile >> goto [note: org-refile with a goto arg, not org-goto] has worked >> perfectly. with no cache. now, there is an issue, where with no >> cache that i know of, the first use, or the first use in a long time, >> will actually present a huge file list that includes crazy elements >> and is not constrained by even the verify function. > > Concerning performance. After reading your message I have realized that > it should be really fast to extract several thousand headings from a > buffer using regexp. However actually it is not so: > > (benchmark-run 10 (and (org-refile-get-targets) nil)) > | 9.868742533999999 | 40 | 2.999932755999996 | > > So preparing the list of refile targets currently takes almost precisely > 1 second. It is slow. Results are same for org-9.1.6 and > release_9.4.4-231-gf46925. I am surprised however that 9.1.6 and 9.3.1 > are installed as system packages and have compiled files. I have not > compiled git version but it works with the same speed. > > (length (org-refile-get-targets)) > : 3220 > > Preferences: > > (require 'org-refile) > (setq org-agenda-files '("~/notes/notes.org")) > (setq org-refile-use-cache nil) > (setq org-refile-use-outline-path t) > (setq org-outline-path-complete-in-steps nil) > (setq org-refile-targets '((org-agenda-files :maxlevel . 5))) > > Some lines from profile: > > - org-refile-get-targets 8414 74% > ... - org-get-outline-path 7403 65% > ... - org--get-outline-path-1 7286 64% > ... - org-up-heading-safe 6328 56% > > There is a room for improvement. Outline paths could be obtained in a > single pass without backward search. It should speed up building the > list of targets by 2 or 3 times. > > Though it is unrelated to issues with default option and cache entries > when both ways to call org-refile are used: directly and through org-goto. > > > -- The Kafka Pandemic Please learn what misopathy is. https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html
if you are willing to try something as a complete stab in the dark, here is something i have had from many years ago that fixed issues possibly including a defaulting issue. the issues include other stuff too, but i do not understand the code now for health reasons to rule this out as a solution to your problem in addition to the problem i put in the comment and name. i havce carried along this patch for eyars. commit deaa14b6ed264c259a1f3b805b67b2db3951ba20 Author: Your Name <you@example.com> Date: 2020-09-20 13:30:55 -0700 === alpha remove the parens from ido completion of olpaths Modified lisp/org-refile.el diff --git a/lisp/org-refile.el b/lisp/org-refile.el index 1e6872b46..8f98e9cf9 100644 --- a/lisp/org-refile.el +++ b/lisp/org-refile.el @@ -617,7 +617,9 @@ this function appends the default value from (tbl (mapcar (lambda (x) (if (and (not (member org-refile-use-outline-path - '(file full-file-path))) + ;; === alpha remove the parens from ido completion of olpaths + '(nil full-file-path))) + ;; '(file full-file-path))) (not (equal filename (nth 1 x)))) (cons (concat (car x) extra " (" (file-name-nondirectory (nth 1 x)) ")") On 3/4/21, Maxim Nikulin <manikulin@gmail.com> wrote: > On 03/03/2021 00:15, Maxim Nikulin wrote: >> >> There is one issue however. Default option option does not work if after >> cache clean other command is called, e.g. >> - jump using C-u C-c C-j >> - clean cache C-u C-u C-u C-c C-w >> - try to jump or to refile [C-u] C-c C-w to default offered option >> - "user-error: Invalid target location" > > I have realized that this issue is rather loosely related to > org-refile-use-cache and I have seen it in a worse variant with disabled > cache. > > file init-refile.el: > > (package-initialize) > > (custom-set-variables > '(org-agenda-files (quote ("~/examples/org/test-notes.org"))) > '(org-capture-templates > (quote > (("t" "Test" entry > (file "") > "* %?\n\n%U\n" :empty-lines 1)))) > '(org-default-notes-file "~/examples/org/capture.org") > '(org-directory "~/examples/org/") > '(org-modules (quote (org-refile))) > '(org-outline-path-complete-in-steps nil) > '(org-refile-targets (quote ((nil :maxlevel . 5) (org-agenda-files > :maxlevel . 5)))) > '(org-refile-use-outline-path t) > ) > > emacs -Q -L ~/src/emacs/org-mode/lisp/ -L > ~/src/emacs/org-mode/contrib/lisp/ -l ~/examples/org/init-refile.el > ~/examples/org/test-notes.org > > - C-u C-c C-j (org-goto) to some heading in the test-notes.org, e.g. > "Two" / "Third" / "Theme" > - M-x org-capture RET t to capture some note > - Let's assume that target of previously executed org-goto suits for > this note > - C-c C-w RET to refile the note from capture.org file to the heading in > test-notes.org offered as the default option. > > Actual result: > - "Invalid target location" > - Captured note is still in the capture.org file but it is not apparent > since capture frame is closed. > > Expected result: > - Default option works for refile even if it is remained from org-goto > command > > Side note. In some cases it not so easy to close capture frame. If > org-default-note-file is configured to a file in a non-existing > directory then C-c C-k asks to create that directory but does not > discards the note. > > > -- The Kafka Pandemic Please learn what misopathy is. https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html
[-- Attachment #1: Type: text/plain, Size: 705 bytes --] On 05/03/2021 04:03, Samuel Wales wrote: > interesting. that would be great to speed it up. [i just meant that > the file list used to be correct.] I am a bit disappointed. I have managed to get x2 performance boost. At first, the result was x2 better but I realized that I did not added heading cleanup for the new strategy. It adds several regexp to the inner loop and severely hits performance. I do not like the idea to manually inline (with minor variations) existing functions and regexps, but the function is still slow. For a while, improvement is significant, so I am attaching the patch that should make jumps using org-goto or org-refile faster. I hope, I have not broken anything. [-- Attachment #2: refile-get-targets-outline-path-tests.patch --] [-- Type: text/x-patch, Size: 2012 bytes --] commit 45cfa5b15e9009fee4f6a688caa210ff543b1ac1 Author: Max Nikulin <manikulin@gmail.com> Date: Sat Mar 6 22:14:39 2021 +0700 testing/lisp/test-org.el: More tests for `org-refile-get-targets' testing/lisp/test-org.el (test-org/refile-get-targets): Add a few more cases for the `org-refile-get-targets' function. - Fraction of completed subtasks is removed from heading. - `:level' filter ignores headings having over levels. - Outline path works with `:tag' filter. The aim is to increase coverage for experiments with optimizing of the function. diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el index 2d727ba7a..da313b45b 100644 --- a/testing/lisp/test-org.el +++ b/testing/lisp/test-org.el @@ -6369,6 +6369,27 @@ Paragraph<point>" (let ((org-refile-use-outline-path t) (org-refile-targets `((nil :maxlevel . 1)))) (mapcar #'car (org-refile-get-targets)))))) + ;; When providing targets as paths, clean fraction cookie. + (should + (equal '("H1") + (org-test-with-temp-text "* H1 [1/1]" + (let ((org-refile-use-outline-path t) + (org-refile-targets '((nil :maxlevel . 1)))) + (mapcar #'car (org-refile-get-targets)))))) + ;; When providing targets as paths, intermediate paths are cached correctly. + (should + (equal '("H1/H2") + (org-test-with-temp-text "* Skip 1\n* H1\n*** Skip 2\n** H2\n*** Skip 3" + (let ((org-refile-use-outline-path t) + (org-refile-targets '((nil :level . 2)))) + (mapcar #'car (org-refile-get-targets)))))) + ;; When providing targets as paths, they are obtained correctly. + (should + (equal '("H1/H2" "H3") + (org-test-with-temp-text "* Skip 1\n* H1\n** Skip 2\n** H2 :take:\n* H3 :take:" + (let ((org-refile-use-outline-path t) + (org-refile-targets '((nil :tag . "take")))) + (mapcar #'car (org-refile-get-targets)))))) ;; When `org-refile-use-outline-path' is `file', include file name ;; without directory in targets. (should [-- Attachment #3: refile-get-targets-outline-path-optimize.patch --] [-- Type: text/x-patch, Size: 4030 bytes --] commit 58b477e999f3bb5b48c39fe0a4e5ad0d37e2bb9d Author: Max Nikulin <manikulin@gmail.com> Date: Sat Mar 6 22:44:27 2021 +0700 lisp/org-refile.el: Speed up `org-refile-get-targets' lisp/org-refile.el (org-refile-get-targets): Optimize performance by eliminating backward lookup of already seen headers. If configuration allows it, incrementally update current outline path. For dense target trees (`:maxlevel' and `:level') it allows to avoid "one step forward, two steps back" strategy that requires multiple backward searches for deeply nested headings. diff --git a/lisp/org-refile.el b/lisp/org-refile.el index 4e9f26eff..8e760f1c3 100644 --- a/lisp/org-refile.el +++ b/lisp/org-refile.el @@ -267,7 +267,8 @@ converted to a headline before refiling." (let ((case-fold-search nil) ;; otherwise org confuses "TODO" as a kw and "Todo" as a word (entries (or org-refile-targets '((nil . (:level . 1))))) - targets tgs files desc descre) + targets tgs files desc descre + outline-path cache-outline-path target-outline-level) (message "Getting targets...") (with-current-buffer (or default-buffer (current-buffer)) (dolist (entry entries) @@ -281,6 +282,11 @@ converted to a headline before refiling." ((and (symbolp files) (boundp files)) (setq files (symbol-value files)))) (when (stringp files) (setq files (list files))) + (setq cache-outline-path (and org-refile-use-outline-path + (memq (car desc) '(:level :maxlevel)))) + (setq target-outline-level + (if (and cache-outline-path (eq (car desc) :level)) + (if org-odd-levels-only (1- (* 2 (cdr desc))) (cdr desc)))) (cond ((eq (car desc) :tag) (setq descre (concat "^\\*+[ \t]+.*?:" (regexp-quote (cdr desc)) ":"))) @@ -288,13 +294,13 @@ converted to a headline before refiling." (setq descre (concat "^\\*+[ \t]+" (regexp-quote (cdr desc)) "[ \t]"))) ((eq (car desc) :regexp) (setq descre (cdr desc))) - ((eq (car desc) :level) + ((and (not target-outline-level) (eq (car desc) :level)) (setq descre (concat "^\\*\\{" (number-to-string (if org-odd-levels-only (1- (* 2 (cdr desc))) (cdr desc))) "\\}[ \t]"))) - ((eq (car desc) :maxlevel) + ((memq (car desc) '(:level :maxlevel)) (setq descre (concat "^\\*\\{1," (number-to-string (if org-odd-levels-only (1- (* 2 (cdr desc))) @@ -318,13 +324,30 @@ converted to a headline before refiling." (org-with-wide-buffer (goto-char (point-min)) (setq org-outline-path-cache nil) + (setq outline-path nil) (while (re-search-forward descre nil t) (beginning-of-line) (let ((case-fold-search nil)) (looking-at org-complex-heading-regexp)) (let ((begin (point)) - (heading (match-string-no-properties 4))) - (unless (or (and + (heading (match-string-no-properties 4)) + (heading-level (length (match-string-no-properties 1)))) + (when cache-outline-path + (while (and outline-path (<= heading-level (caar outline-path))) + (pop outline-path)) + (push (cons heading-level + ;; Taken from org--get-outline-path-1. It is really slow. + (if (not heading) + "" + (org-trim + (org-link-display-format + (replace-regexp-in-string + "\\[[0-9]+%\\]\\|\\[[0-9]+/[0-9]+\\]" "" + heading))))) + outline-path)) + (unless (or (and target-outline-level + (not (eq heading-level target-outline-level))) + (and org-refile-target-verify-function (not (funcall org-refile-target-verify-function))) @@ -349,7 +372,9 @@ converted to a headline before refiling." (_ nil)) (mapcar (lambda (s) (replace-regexp-in-string "/" "\\/" s nil t)) - (org-get-outline-path t t))) + (if outline-path + (nreverse (mapcar #'cdr outline-path)) + (org-get-outline-path t t)))) "/")))) (push (list target f re (org-refile-marker (point))) tgs)))
On 05/03/2021 05:53, Samuel Wales wrote:
> diff --git a/lisp/org-refile.el b/lisp/org-refile.el
> index 1e6872b46..8f98e9cf9 100644
> --- a/lisp/org-refile.el
> +++ b/lisp/org-refile.el
> @@ -617,7 +617,9 @@ this function appends the default value from
> (tbl (mapcar
> (lambda (x)
> (if (and (not (member org-refile-use-outline-path
> - '(file full-file-path)))
> + ;; === alpha remove the parens
> from ido completion of olpaths
> + '(nil full-file-path)))
> + ;; '(file full-file-path)))
> (not (equal filename (nth 1 x))))
> (cons (concat (car x) extra " ("
> (file-name-nondirectory (nth 1 x)) ")")
The patch changes appearance of "(file.org)" at the end of each
completion option. I do not know value of your
org-refile-use-outline-path setting so I am unsure if your are trying to
avoid it or vice versa to force it.
I do not think that the patch is relevant to sharing of default option
between org-goto and org-refile. I think, buffer (or file) name should
be stored separately from outline path within the buffer in org-refile
history and merged accordingly to current settings just before
completing-read. org-refile-cache likely requires a similar approach.
Personally I was confused by behavior of (org-refile-use-outline-path
t). Completion options appears as "A (file.org)", "A/B (file.org)", and
I could not figure out how to refile directly to "A" headings. I suspect
that org-refile-allow-creating-parent-nodes does not work with "t" at
all. Recently I have realized that after typing of "A/B" TAB completes
option to "A/B (file.org)" and I was trying RET after "A". It is
necessary to add space "A " to refile to a non-leaf node.
That was the reason why I set (org-refile-use-outline-path 'file).
Completion options appears as "file.org/A/B". I have to press TAB at
first to get "file.org/" but then after "file.org/A" RET works fine.
Hi Maxim,
Maxim Nikulin <manikulin@gmail.com> writes:
> For a while, improvement is significant, so I am attaching the patch
> that should make jumps using org-goto or org-refile faster. I hope, I
> have not broken anything.
Thanks a lot for the patch.
Are you still using it? Did anyone tested it?
On 25/04/2021 19:25, Bastien wrote:
>
> Are you still using it? Did anyone tested it?
I realized that I break cache a bit, so the patch should not be applied
in the current form. I have an idea how to fix it but I have found more
problems around (independent of my patch). I did not worried since the
patch did not appeared on updates.orgmode.org (I guess, due to "git
diff" instead of "git format patch".)
Cache key is regexp. With my patch, level and max-level options share
the same regexp. I think, cache for both options could be generated as
for max-level and filtered in the case of "level". It may simplify the code.
I would be grateful for other comments.
I have not realized how to reduce number of regexps inside the loop
without duplication of some regegexps. I have seen a statement that
cache for compiled regular expressions in emacs is rather small so
several patterns used on each cycle is enough to make it useless with
significant performance penalty.
Other issues:
- Indirect buffer created with C-x 4 c does not reuse cache created for
the main buffer, so first call in new buffer have to generate its own cache.
- C-u C-c C-j promises to use headings from the buffer, but in the case
of subtree indirect buffer, headings are not filtered to the narrowed part.
- I have not solved the problem with default option for `org-refile' and
`org-goto'. Maybe more detailed structure should be stored in history
and formatted accordingly to current option at the moment of invocation.