Org-mode mailing list
 help / color / mirror / Atom feed
From: Ihor Radchenko <yantar92@gmail.com>
To: Christian Heinrich <christian@gladbachcity.de>, emacs-orgmode@gnu.org
Subject: Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers
Date: Sat, 09 May 2020 23:46:08 +0800
Message-ID: <87blmxw1q7.fsf@localhost> (raw)
In-Reply-To: <78ccc02ac16d47c97766865fdf5206df57023444.camel@gladbachcity.de>

> I am not sure I understand how your follow-up code (below) needs to be incorporated. Would you mind
> sending a patch file? I hope that this ends up in the master branch at some point.

I have sent the patch in another email.
Will appreciate any feedback.

Best,
Ihor


Christian Heinrich <christian@gladbachcity.de> writes:

> Hi,
>
> thanks for your (initial) patch! I traced another error down today and found your code by chance. I
> tested it on an org-drill file that I had (with over 3500 items and hence 3500 drawers) and this
> patch helps *a lot* already. (Performance broke in 4403d4685e19fb99ba9bfec2bd4ff6781c66981f when
> outline-flag-region was replaced with org-flag-region, as drawers are no longer opened using
> outline-show-all which I had to use anyways to deal with my huge file.)
>
> I am not sure I understand how your follow-up code (below) needs to be incorporated. Would you mind
> sending a patch file? I hope that this ends up in the master branch at some point.
>
> Thanks again!
> Christian
>
> On Mon, 2020-04-27 at 00:04 +0800, Ihor Radchenko wrote:
>> > You cannot. You may however mimic it with `cursor-sensor-functions' text
>> > property. These assume Cursor Sensor minor mode is active, tho.
>> > I haven't tested it, but I assume it would slow down text properties
>> > a bit, too, but hopefully not as much as overlays.
>> 
>> Unfortunately, isearch sets inhibit-point-motion-hooks to non-nil
>> internally. Anyway, I came up with some workaround, which seems to work
>> (see below). Though it would be better if isearch supported hidden text
>> in addition to overlays.
>> 
>> > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth
>> > experimenting with `cursor-sensor-functions'.
>> 
>> So far, I came up with the following partial solution searching and
>> showing hidden text.
>> 
>> ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we
>> ;; cannot even use cursor-sensor-functions as a workaround
>> ;; I used a less ideas approach with advice to isearch-search-string as
>> ;; a workaround 
>> 
>> (defun org-find-text-property-region (pos prop)
>>   "Find a region containing PROP text property around point POS."
>>   (require 'org-macs) ;; org-with-point-at
>>   (org-with-point-at pos
>>     (let* ((beg (and (get-text-property pos prop) pos))
>> 	   (end beg))
>>       (when beg
>> 	(setq beg (or (previous-single-property-change pos prop)
>> 		      beg))
>> 	(setq end (or (next-single-property-change pos prop)
>> 		      end))
>>         (unless (equal beg end)
>>           (cons beg end))))))
>> 
>> ;; :FIXME: re-hide properties when point moves away
>> (define-advice isearch-search-string (:after (&rest _) put-overlay)
>>   "Reveal hidden text at point."
>>   (when-let ((region (org-find-text-property-region (point) 'invisible)))
>>     (with-silent-modifications
>>       (put-text-property (car region) (cdr region) 'org-invisible (get-text-property (point)
>> 'invisible)))
>>       (remove-text-properties (car region) (cdr region) '(invisible nil))))
>> 
>> ;; this seems to be unstable, but I cannot figure out why
>> (defun org-restore-invisibility-specs (&rest _)
>>   ""
>>    (let ((pos (point-min)))
>>      (while (< (setq pos (next-single-property-change pos 'org-invisible nil (point-max))) (point-
>> max))
>>        (when-let ((region (org-find-text-property-region pos 'org-invisible)))
>> 	   (with-silent-modifications
>> 	     (put-text-property (car region) (cdr region) 'invisible (get-text-property pos 'org-
>> invisible))
>> 	     (remove-text-properties (car region) (cdr region) '(org-invisible nil)))))))
>> 
>> (add-hook 'post-command-hook #'org-restore-invisibility-specs)
>> 
>> (defun org-flag-region (from to flag spec)
>>   "Hide or show lines from FROM to TO, according to FLAG.
>> SPEC is the invisibility spec, as a symbol."
>>   (pcase spec
>>     ('outline
>>      (remove-overlays from to 'invisible spec)
>>      ;; Use `front-advance' since text right before to the beginning of
>>      ;; the overlay belongs to the visible line than to the contents.
>>      (when flag
>>        (let ((o (make-overlay from to nil 'front-advance)))
>> 	 (overlay-put o 'evaporate t)
>> 	 (overlay-put o 'invisible spec)
>> 	 (overlay-put o 'isearch-open-invisible #'delete-overlay))))
>>     (_
>>      (with-silent-modifications
>>        (remove-text-properties from to '(invisible nil))
>>        (when flag
>> 	 (put-text-property from to 'invisible spec)
>> 	 )))))
>> 
>> ;; This normally deletes invisible text property. We do not want this now.
>> (defun org-unfontify-region (beg end &optional _maybe_loudly)
>>   "Remove fontification and activation overlays from links."
>>   (font-lock-default-unfontify-region beg end)
>>   (let* ((buffer-undo-list t)
>> 	 (inhibit-read-only t) (inhibit-point-motion-hooks t)
>> 	 (inhibit-modification-hooks t)
>> 	 deactivate-mark buffer-file-name buffer-file-truename)
>>     (decompose-region beg end)
>>     (remove-text-properties beg end
>> 			    '(mouse-face t keymap t org-linked-text t
>> 					 ;; Do not remove invisible during fontification			
>> 		 
>> 					 ;; invisible t
>>                                          intangible t
>> 					 org-emphasis t))
>>     (org-remove-font-lock-display-properties beg end)))
>> 
>> > Anyway, the real fix should come from Emacs itself. There are ways to
>> > make overlays faster. These ways have already been discussed on the
>> > Emacs devel mailing list, but no one implemented them. It is a bit sad
>> > that we have to find workarounds for that.
>> 
>> I guess that it is a very old story starting from the times when XEmacs
>> was a thing [1]. I recently heard about binary tree implementation of
>> overlays (there should be a branch in emacs git repo) [2], but there was
>> no update on that branch for a while. So, I do not have much hope on
>> Emacs implementing efficient overlay access in the near future. (And I
>> have problems with huge org files already).
>> 
>> [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid_emacs_fsf_emacs_schism/
>> [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html
>> 
>> 
>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>> 
>> > Hello,
>> > 
>> > Ihor Radchenko <yantar92@gmail.com> writes:
>> > 
>> > > To my surprise, the patch did not break org to unusable state and
>> > > the performance on the sample org file [3] improved drastically. You can
>> > > try by yourself!
>> > 
>> > It is not a surprise, really. Text properties are much faster than
>> > overlays, and very close to them features-wise. They are a bit more
>> > complex to handle, however.
>> > 
>> > > However, this did introduce some visual glitches with drawer display.
>> > > Though drawers can still be folded/unfolded with <tab>, they are not
>> > > folded on org-mode startup for some reason (can be fixed by running
>> > > (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawers)
>> > > are unfolded for no apparent reason sometimes. A blind guess is that it
>> > > is something to do with lack of 'isearch-open-invisible, which I am not
>> > > sure how to set via text properties.
>> > 
>> > You cannot. You may however mimic it with `cursor-sensor-functions' text
>> > property. These assume Cursor Sensor minor mode is active, tho.
>> > I haven't tested it, but I assume it would slow down text properties
>> > a bit, too, but hopefully not as much as overlays.
>> > 
>> > Note there are clear advantages using text properties. For example, when
>> > you move contents around, text properties are preserved. So there's no
>> > more need for the `org-cycle-hide-drawer' dance, i.e., it is not
>> > necessary anymore to re-hide drawers.
>> > 
>> > > Any thoughts about the use of text properties or about the patch
>> > > suggestion are welcome.  
>> > 
>> > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be worth
>> > experimenting with `cursor-sensor-functions'.
>> > 
>> > We could also use text properties for property drawers, and overlays for
>> > regular ones. This might give us a reasonable speed-up with an
>> > acceptable feature trade-off.
>> > 
>> > Anyway, the real fix should come from Emacs itself. There are ways to
>> > make overlays faster. These ways have already been discussed on the
>> > Emacs devel mailing list, but no one implemented them. It is a bit sad
>> > that we have to find workarounds for that.
>> > 
>> > Regards,
>> > 
>> > -- 
>> > Nicolas Goaziou

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg


  reply	other threads:[~2020-05-09 15:50 UTC|newest]

Thread overview: 85+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-04-24  6:55 Ihor Radchenko
2020-04-24  8:02 ` Nicolas Goaziou
2020-04-25  0:29   ` stardiviner
2020-04-26 16:04   ` Ihor Radchenko
2020-05-04 16:56     ` Karl Voit
2020-05-07  7:18       ` Karl Voit
2020-05-09 15:43       ` Ihor Radchenko
2020-05-07 11:04     ` Christian Heinrich
2020-05-09 15:46       ` Ihor Radchenko [this message]
2020-05-08 16:38     ` Nicolas Goaziou
2020-05-09 13:58       ` Nicolas Goaziou
2020-05-09 16:22         ` Ihor Radchenko
2020-05-09 17:21           ` Nicolas Goaziou
2020-05-10  5:25             ` Ihor Radchenko
2020-05-10  9:47               ` Nicolas Goaziou
2020-05-10 13:29                 ` Ihor Radchenko
2020-05-10 14:46                   ` Nicolas Goaziou
2020-05-10 16:21                     ` Ihor Radchenko
2020-05-10 16:38                       ` Nicolas Goaziou
2020-05-10 17:08                         ` Ihor Radchenko
2020-05-10 19:38                           ` Nicolas Goaziou
2020-05-09 15:40       ` Ihor Radchenko
2020-05-09 16:30         ` Ihor Radchenko
2020-05-09 17:32           ` Nicolas Goaziou
2020-05-09 18:06             ` Ihor Radchenko
2020-05-10 14:59               ` Nicolas Goaziou
2020-05-10 15:15                 ` Kyle Meyer
2020-05-10 16:30                 ` Ihor Radchenko
2020-05-10 19:32                   ` Nicolas Goaziou
2020-05-12 10:03                     ` Nicolas Goaziou
2020-05-17 15:00                     ` Ihor Radchenko
2020-05-17 15:40                       ` Ihor Radchenko
2020-05-18 14:35                         ` Nicolas Goaziou
2020-05-18 16:52                           ` Ihor Radchenko
2020-05-19 13:07                             ` Nicolas Goaziou
2020-05-23 13:52                               ` Ihor Radchenko
2020-05-23 13:53                                 ` Ihor Radchenko
2020-05-23 15:26                                   ` Ihor Radchenko
2020-05-26  8:33                                 ` Nicolas Goaziou
2020-06-02  9:21                                   ` Ihor Radchenko
2020-06-02  9:23                                     ` Ihor Radchenko
2020-06-02 12:10                                       ` Bastien
2020-06-02 13:12                                         ` Ihor Radchenko
2020-06-02 13:23                                           ` Bastien
2020-06-02 13:30                                             ` Ihor Radchenko
2020-06-02  9:25                                     ` Ihor Radchenko
2020-06-05  7:26                                     ` Nicolas Goaziou
2020-06-05  8:18                                       ` Ihor Radchenko
2020-06-05 13:50                                         ` Nicolas Goaziou
2020-06-08  5:05                                           ` Ihor Radchenko
2020-06-08  5:06                                             ` Ihor Radchenko
2020-06-08  5:08                                             ` Ihor Radchenko
2020-06-10 17:14                                             ` Nicolas Goaziou
2020-06-21  9:52                                               ` Ihor Radchenko
2020-06-21 15:01                                                 ` Nicolas Goaziou
2020-08-11  6:45                                               ` Ihor Radchenko
2020-08-11 23:07                                                 ` Kyle Meyer
2020-08-12  6:29                                                   ` Ihor Radchenko
2020-09-20  5:53                                                     ` Ihor Radchenko
2020-09-20 11:45                                                       ` Kévin Le Gouguec
2020-09-22  9:05                                                         ` Ihor Radchenko
2020-09-22 10:00                                                           ` Ihor Radchenko
2020-09-23  6:16                                                             ` Kévin Le Gouguec
2020-09-23  6:48                                                               ` Ihor Radchenko
2020-09-23  7:09                                                                 ` Bastien
2020-09-23  7:30                                                                   ` Ihor Radchenko
2020-09-24 18:07                                                                 ` Kévin Le Gouguec
2020-09-25  2:16                                                                   ` Ihor Radchenko
2020-12-15 17:38                                                                     ` [9.4] Fixing logbook visibility during isearch Kévin Le Gouguec
2020-12-16  3:15                                                                       ` Ihor Radchenko
2020-12-16 18:05                                                                         ` Kévin Le Gouguec
2020-12-17  3:18                                                                           ` Ihor Radchenko
2020-12-17 14:50                                                                             ` Kévin Le Gouguec
2020-12-18  2:23                                                                               ` Ihor Radchenko
2020-12-24 23:37                                                                                 ` Kévin Le Gouguec
2020-12-25  2:51                                                                                   ` Ihor Radchenko
2020-12-25 10:59                                                                                     ` Kévin Le Gouguec
2020-12-25 12:32                                                                                       ` Ihor Radchenko
2020-12-25 21:35                                                                                     ` Kévin Le Gouguec
2020-12-26  4:14                                                                                       ` Ihor Radchenko
2020-12-26 11:44                                                                                         ` Kévin Le Gouguec
2020-12-26 12:22                                                                                           ` Ihor Radchenko
2020-12-04  5:58                                                       ` [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers Ihor Radchenko
2021-03-21  9:09                                                         ` Ihor Radchenko
2021-05-03 17:28                                                           ` Bastien

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://orgmode.org

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87blmxw1q7.fsf@localhost \
    --to=yantar92@gmail.com \
    --cc=christian@gladbachcity.de \
    --cc=emacs-orgmode@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Org-mode mailing list

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://orgmode.org/list/0 list/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 list list/ https://orgmode.org/list \
		emacs-orgmode@gnu.org
	public-inbox-index list

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.emacs.orgmode
	nntp://news.gmane.io/gmane.emacs.orgmode


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git