From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id ON5oMJ3rs17XNwAA0tVLHw (envelope-from ) for ; Thu, 07 May 2020 11:06:05 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id KCZfOanrs14zBQAAB5/wlQ (envelope-from ) for ; Thu, 07 May 2020 11:06:17 +0000 Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:470:142::17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id BE03694005C for ; Thu, 7 May 2020 11:06:14 +0000 (UTC) Received: from localhost ([::1]:48328 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jWeM2-0001EE-HD for larch@yhetil.org; Thu, 07 May 2020 07:06:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52710) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jWeLD-0001CY-7O for emacs-orgmode@gnu.org; Thu, 07 May 2020 07:05:23 -0400 Received: from gallois.livando.com ([31.170.109.33]:41684) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jWeLA-0006eB-Va for emacs-orgmode@gnu.org; Thu, 07 May 2020 07:05:22 -0400 X-PDA-ORIGIN: gallois.livando.com Received: (qmail 7564 invoked from network); 7 May 2020 11:04:16 -0000 Received: by simscan 1.4.0 ppid: 7558, pid: 7561, t: 0.1284s scanners:none Received: from unknown (HELO cantor.fritz.box) (christian@gladbachcity.de@87.78.250.7) by 0 with SMTP; 7 May 2020 11:04:16 -0000 Message-ID: <78ccc02ac16d47c97766865fdf5206df57023444.camel@gladbachcity.de> Subject: Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers From: Christian Heinrich To: emacs-orgmode@gnu.org Date: Thu, 07 May 2020 13:04:15 +0200 In-Reply-To: <87y2qi8c8w.fsf@localhost> References: <87h7x9e5jo.fsf@localhost> <875zdpia5i.fsf@nicolasgoaziou.fr> <87y2qi8c8w.fsf@localhost> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-Kn0RvlLG6W1BlkWOlTNP" User-Agent: Evolution 3.36.1-1 MIME-Version: 1.0 Received-SPF: none client-ip=31.170.109.33; envelope-from=christian@gladbachcity.de; helo=gallois.livando.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/07 07:05:16 X-ACL-Warn: Detected OS = Linux 3.x [generic] X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Scanner: scn0 X-Spam-Score: -3.11 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 2001:470:142::17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Scan-Result: default: False [-3.11 / 13.00]; GENERIC_REPUTATION(0.00)[-0.49774842981328]; DWL_DNSWL_FAIL(0.00)[2001:470:142::17:server fail]; IP_REPUTATION_HAM(0.00)[asn: 22989(0.11), country: US(-0.00), ip: 2001:470:142::17(-0.50)]; R_SPF_ALLOW(-0.20)[+ip6:2001:470:142::/48]; TO_DN_NONE(0.00)[]; MX_GOOD(-0.50)[eggs.gnu.org]; MAILLIST(-0.20)[mailman]; SIGNED_PGP(-2.00)[]; FORGED_RECIPIENTS_MAILLIST(0.00)[]; RCVD_IN_DNSWL_FAIL(0.00)[2001:470:142::17:server fail]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:22989, ipnet:2001:470:142::/48, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TAGGED_FROM(0.00)[larch=yhetil.org]; FROM_NEQ_ENVFROM(0.00)[christian@gladbachcity.de,emacs-orgmode-bounces@gnu.org]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; SPF_REPUTATION_HAM(0.00)[-0.57055930810528]; URIBL_BLOCKED(0.00)[nicolasgoaziou.fr:email,gnu.org:url,reddit.com:url]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[gladbachcity.de]; HAS_LIST_UNSUB(-0.01)[]; RCPT_COUNT_ONE(0.00)[1]; FORGED_SENDER_MAILLIST(0.00)[] X-TUID: JeVRiKtrj7om --=-Kn0RvlLG6W1BlkWOlTNP Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, thanks for your (initial) patch! I traced another error down today and foun= d your code by chance. I tested it on an org-drill file that I had (with over 3500 items and hence 3= 500 drawers) and this patch helps *a lot* already. (Performance broke in 4403d4685e19fb99ba9bfec2= bd4ff6781c66981f when outline-flag-region was replaced with org-flag-region, as drawers are no lo= nger opened using outline-show-all which I had to use anyways to deal with my huge file.) I am not sure I understand how your follow-up code (below) needs to be inco= rporated. Would you mind sending a patch file? I hope that this ends up in the master branch at some= point. Thanks again! Christian On Mon, 2020-04-27 at 00:04 +0800, Ihor Radchenko wrote: > > You cannot. You may however mimic it with `cursor-sensor-functions' tex= t > > property. These assume Cursor Sensor minor mode is active, tho. > > I haven't tested it, but I assume it would slow down text properties > > a bit, too, but hopefully not as much as overlays. >=20 > Unfortunately, isearch sets inhibit-point-motion-hooks to non-nil > internally. Anyway, I came up with some workaround, which seems to work > (see below). Though it would be better if isearch supported hidden text > in addition to overlays. >=20 > > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be wort= h > > experimenting with `cursor-sensor-functions'. >=20 > So far, I came up with the following partial solution searching and > showing hidden text. >=20 > ;; Unfortunately isearch, sets inhibit-point-motion-hooks and we > ;; cannot even use cursor-sensor-functions as a workaround > ;; I used a less ideas approach with advice to isearch-search-string as > ;; a workaround=20 >=20 > (defun org-find-text-property-region (pos prop) > "Find a region containing PROP text property around point POS." > (require 'org-macs) ;; org-with-point-at > (org-with-point-at pos > (let* ((beg (and (get-text-property pos prop) pos)) > (end beg)) > (when beg > (setq beg (or (previous-single-property-change pos prop) > beg)) > (setq end (or (next-single-property-change pos prop) > end)) > (unless (equal beg end) > (cons beg end)))))) >=20 > ;; :FIXME: re-hide properties when point moves away > (define-advice isearch-search-string (:after (&rest _) put-overlay) > "Reveal hidden text at point." > (when-let ((region (org-find-text-property-region (point) 'invisible))) > (with-silent-modifications > (put-text-property (car region) (cdr region) 'org-invisible (get-te= xt-property (point) > 'invisible))) > (remove-text-properties (car region) (cdr region) '(invisible nil))= )) >=20 > ;; this seems to be unstable, but I cannot figure out why > (defun org-restore-invisibility-specs (&rest _) > "" > (let ((pos (point-min))) > (while (< (setq pos (next-single-property-change pos 'org-invisible = nil (point-max))) (point- > max)) > (when-let ((region (org-find-text-property-region pos 'org-invisib= le))) > (with-silent-modifications > (put-text-property (car region) (cdr region) 'invisible (get-text-p= roperty pos 'org- > invisible)) > (remove-text-properties (car region) (cdr region) '(org-invisible n= il))))))) >=20 > (add-hook 'post-command-hook #'org-restore-invisibility-specs) >=20 > (defun org-flag-region (from to flag spec) > "Hide or show lines from FROM to TO, according to FLAG. > SPEC is the invisibility spec, as a symbol." > (pcase spec > ('outline > (remove-overlays from to 'invisible spec) > ;; Use `front-advance' since text right before to the beginning of > ;; the overlay belongs to the visible line than to the contents. > (when flag > (let ((o (make-overlay from to nil 'front-advance))) > (overlay-put o 'evaporate t) > (overlay-put o 'invisible spec) > (overlay-put o 'isearch-open-invisible #'delete-overlay)))) > (_ > (with-silent-modifications > (remove-text-properties from to '(invisible nil)) > (when flag > (put-text-property from to 'invisible spec) > ))))) >=20 > ;; This normally deletes invisible text property. We do not want this now= . > (defun org-unfontify-region (beg end &optional _maybe_loudly) > "Remove fontification and activation overlays from links." > (font-lock-default-unfontify-region beg end) > (let* ((buffer-undo-list t) > (inhibit-read-only t) (inhibit-point-motion-hooks t) > (inhibit-modification-hooks t) > deactivate-mark buffer-file-name buffer-file-truename) > (decompose-region beg end) > (remove-text-properties beg end > '(mouse-face t keymap t org-linked-text t > ;; Do not remove invisible during fontification =09 > =20 > ;; invisible t > intangible t > org-emphasis t)) > (org-remove-font-lock-display-properties beg end))) >=20 > > Anyway, the real fix should come from Emacs itself. There are ways to > > make overlays faster. These ways have already been discussed on the > > Emacs devel mailing list, but no one implemented them. It is a bit sad > > that we have to find workarounds for that. >=20 > I guess that it is a very old story starting from the times when XEmacs > was a thing [1]. I recently heard about binary tree implementation of > overlays (there should be a branch in emacs git repo) [2], but there was > no update on that branch for a while. So, I do not have much hope on > Emacs implementing efficient overlay access in the near future. (And I > have problems with huge org files already). >=20 > [1] https://www.reddit.com/r/planetemacs/comments/e9lgwn/history_of_lucid= _emacs_fsf_emacs_schism/ > [2] https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00323.html >=20 >=20 > Nicolas Goaziou writes: >=20 > > Hello, > >=20 > > Ihor Radchenko writes: > >=20 > > > To my surprise, the patch did not break org to unusable state and > > > the performance on the sample org file [3] improved drastically. You = can > > > try by yourself! > >=20 > > It is not a surprise, really. Text properties are much faster than > > overlays, and very close to them features-wise. They are a bit more > > complex to handle, however. > >=20 > > > However, this did introduce some visual glitches with drawer display. > > > Though drawers can still be folded/unfolded with , they are not > > > folded on org-mode startup for some reason (can be fixed by running > > > (org-cycle-hide-drawers 'all)). Also, some drawers (or parts of drawe= rs) > > > are unfolded for no apparent reason sometimes. A blind guess is that = it > > > is something to do with lack of 'isearch-open-invisible, which I am n= ot > > > sure how to set via text properties. > >=20 > > You cannot. You may however mimic it with `cursor-sensor-functions' tex= t > > property. These assume Cursor Sensor minor mode is active, tho. > > I haven't tested it, but I assume it would slow down text properties > > a bit, too, but hopefully not as much as overlays. > >=20 > > Note there are clear advantages using text properties. For example, whe= n > > you move contents around, text properties are preserved. So there's no > > more need for the `org-cycle-hide-drawer' dance, i.e., it is not > > necessary anymore to re-hide drawers. > >=20 > > > Any thoughts about the use of text properties or about the patch > > > suggestion are welcome. =20 > >=20 > > Missing `isearch-open-invisible' is a deal breaker, IMO. It may be wort= h > > experimenting with `cursor-sensor-functions'. > >=20 > > We could also use text properties for property drawers, and overlays fo= r > > regular ones. This might give us a reasonable speed-up with an > > acceptable feature trade-off. > >=20 > > Anyway, the real fix should come from Emacs itself. There are ways to > > make overlays faster. These ways have already been discussed on the > > Emacs devel mailing list, but no one implemented them. It is a bit sad > > that we have to find workarounds for that. > >=20 > > Regards, > >=20 > > --=20 > > Nicolas Goaziou --=-Kn0RvlLG6W1BlkWOlTNP Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEE+9lGLw1S8YBUC27di84V01FvzDwFAl6z6y8ACgkQi84V01Fv zDyWYg//Zad82ktpOAaaM1A93ZFRpKm5SDppcW0hFKDDGxpaEYiEGwsFGb2UqL9P m9hMLYcDbBJVgnNQ4ebKT+J5FcJ0p4Y3v+O5VfILwbtbtEXNEdNB4pFWCBJMp/4Z csDuFHWTu2Pmm6bFioPrwyD01Z5/k5ty8rGCIjmWfR6hHHoOtRbfkH1fJ7n7l/ei QN3fnfQvYPkIvx8POggd+xqYJYdxOlcYk2Aafsjpv6v4CVgKEVU7FxNcC2IGUmWw +nbtAJM6elNwRB5j78ZQ4EoiB6sMuRtFwSjbxYFgtSurz5r8MZi6A9WZK/g4B5MC Aag4hI4aKPAQascZ7bH1NFyQb7iGgkO+YvmSFj3b4LDjRO35ZuCqP2Z/Xi1av3es /Ipx6wjw650kCTm1gVdF/ss1H8MVtysaOvwlMyg/3enDQh2UDou5RfAHkxTTXpd+ g8Gil2tc1+11WkvDMa1cZzvo9o3AaaBQp79a9h6B8D/sAPyIaJODBbQcUnKxgcKg IliMzExfW0WUKPbeBHuBkEFUNDumfSke7xSYL24FwilKovdM4otpcXBZS8jj5G6O 9y3Zx7ngLBKapWqMpMW3Q/IpTttXJ8ijkrH5ZzASgPpyJADY0p8ED6W3NG7rpxK/ UnDdr+R4kXvBkXGc9ECbknwfKgCFuFGkh6I7P+yXqV7UXVejPMc= =ZdaU -----END PGP SIGNATURE----- --=-Kn0RvlLG6W1BlkWOlTNP--