Org-mode mailing list
 help / color / mirror / Atom feed
From: Texas Cyberthal <texas.cyberthal@gmail.com>
To: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
Cc: "emacs-orgmode@gnu.org" <emacs-orgmode@gnu.org>
Subject: Re: One vs many directories
Date: Mon, 23 Nov 2020 17:50:13 +0800
Message-ID: <CAMUm490kEZh5GqLxwNqog1pM0OqUG3_bYoLqQnM9v+db=_my8A@mail.gmail.com> (raw)
In-Reply-To: <87r1olfvh4.fsf@web.de>

Hi Dr. Arne,

> The only part that hits performance limits is the agenda.

Well, IIRC your Org Textmind is much smaller than mine.

> My current guess is that the agenta is slow because it has to parse all my 7500 clock entries, and it has to check the Todo states of around 1200 headings.

Ouch.  I'd rather keep a "ramble log" so I can reconstruct an exactly
honest time accounting, with discounts for partial attention, without
worrying about fiddly clockin/outs.  At least when working from home.
If clocking into a work site, that's different, because one can
reasonably bill for the entire time, with minimal clock toggling.

> Did you check against filesystem limits? At 10k entries in a directory typical filesystems start becoming slow. That’s the main reason I see for adding hierarchies.

10k entries in a directory sounds inhumanely unergonomic.  I guess my
biggest flat name directory might eventually reach that size?  In
which case I could just split it in the middle of the alphabet, or
similar solution.

Right now it's only 600.  If I guess a generous growth rate of 2 per
day, times 30 years, that would be an additional 22k.  Sounds
manageable.

Remember there are ways to consolidate entries even in flat "solid
names" directory.  It's advantageous to do so to facilitate isearch
matching.  For example, everyone with the same last name is one
directory.  Ditto everything that starts with the same word or even
prefix.  For example I have a directory called ~Wiki-~ and another
called ~Tru-~ which contains truth, Trudeau and Trump.

Most adults know 20-35k words.  That's not the same as "solid names"
known, but gives a ballpark on human memory size for a similar name
type.  I suspect computers will advance faster than anyone's Textmind
reaches the Dired lag limit.

No, if we are talking about scaling limits, then limits such as buffer
size and Agenda search speed are orders of magnitude more relevant.
Which problems deep tree nesting fixes.

A 10k entry directory is getting into enterprise territory, and I'm
sure enterprise has tech tricks that become worthwhile at that scale.

> There are scaling problems in every direction: Too many files per directory, too large files, too much content per heading, too many headings.

There are scaling problems from too much deep tree nesting, namely too
much fiddly ambiguous manual refiling.  Solution is flat "solid name"
directories just below feasible 10 Bins.  Work fine.

> I would have to build lots of additional tooling to make that work as well. Many of the tools in Emacs work better on large files than on many files — I will switch to more files when performance on large files reaches its limits.

Nah, my 100 mb (non archived) Textmind works fine.  I just separated
Agenda metadata from bulk prose.

I am curious how many headings I have, how would I count that recursively?

On Sun, Nov 22, 2020 at 8:04 PM Dr. Arne Babenhauserheide
<arne_bab@web.de> wrote:
>
>
> Texas Cyberthal <texas.cyberthal@gmail.com> writes:
>
> >> I need instant search in the knowledge database and quick filing of tasks. Also I need the agenda to create a clocktable (that’s on the limit of being too slow) and the calendar and tasks of the week.
> >
> >> Also I need quick filing of notes and quotes (in specific files, not part of the agenda) and of long-form articles, one file per article (using journal.el, also outside the agenda, searched using M-x deft), and quick creation of website entries for a given category within the site (i.e. M-x draketo-software).
> >
> > So your Org usage style quickly hits critical performance problems at scale.
>
> The only part that hits performance limits is the agenda. All the rest
> scales nicely. My current guess is that the agenta is slow because it
> has to parse all my 7500 clock entries, and it has to check the TODO
> states of around 1200 headings. Having multiple files would only add to
> that.
>
> > I don't have these problems.  Treefactor refiling is immune to scale.
>
> Did you check against filesystem limits? At 10k entries in a directory
> typical filesystems start becoming slow. That’s the main reason I see
> for adding hierarchies.
>
> > Org's many tools and tricks are still handy in niche cases, but they
> > don't cause scaling problems because they don't handle bulk info
> > management.  For example Org's refile tools are useful when writing
> > advanced documentation with large single-file outlines.  Most info
> > doesn't require that much organization.  It works fine as flat lists
> > of headings in a detailed directory tree.
>
> Or as sub-headings in a large outline.
>
> There are scaling problems in every direction: Too many files per
> directory, too large files, too much content per heading, too many
> headings.
>
> I would have to build lots of additional tooling to make that work as
> well. Many of the tools in Emacs work better on large files than on many
> files — I will switch to more files when performance on large files
> reaches its limits.
>
> I have one file where I’m reaching the limit. That’ my 7.3 MiB
> emacs-remember-mode.org file where I throw long-form articles for
> full-text search. I am considering to switch to a multi-file approach
> for that and then to use deft to retrieve articles.
>
> Best wishes,
> Arne
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken


  parent reply	other threads:[~2020-11-23  9:51 UTC|newest]

Thread overview: 151+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-21  0:33 Texas Cyberthal
2020-11-21  5:13 ` Ihor Radchenko
2020-11-21  7:56   ` Jean Louis
2020-11-21  8:31     ` Texas Cyberthal
2020-11-21  9:29       ` Marvin ‘quintus’ Gülker
2020-11-21 10:21       ` Jean Louis
2020-11-21 15:00         ` Texas Cyberthal
2020-11-21 16:08           ` Jean Louis
2020-11-21 15:03     ` Dr. Arne Babenhauserheide
2020-11-21 15:45       ` Texas Cyberthal
2020-11-21 17:12         ` Jean Louis
2020-11-21 18:01           ` Texas Cyberthal
2020-11-21 18:57             ` Jean Louis
2020-11-22  6:36           ` Ihor Radchenko
2020-11-22  7:20             ` Jean Louis
2020-11-22  8:32               ` Ihor Radchenko
2020-11-22  8:56                 ` Jean Louis
2020-11-21 22:36         ` Dr. Arne Babenhauserheide
     [not found]           ` <CAMUm491Psp0u5JKyGROP6M=UfAcvOLTtOKAD1rOearV+KxgYdQ@mail.gmail.com>
     [not found]             ` <87r1olfvh4.fsf@web.de>
2020-11-23  9:50               ` Texas Cyberthal [this message]
2020-11-23 13:17                 ` Jean Louis
2020-11-23 14:16                   ` Ihor Radchenko
2020-11-23 18:08                     ` Is Org really so simple? Jean Louis
2020-11-23 20:41                       ` Tom Gillespie
2020-11-24  5:06                         ` Jean Louis
2020-11-26  3:08                       ` Ihor Radchenko
2020-11-26  8:57                         ` Jean Louis
2020-11-29  7:20                           ` Ihor Radchenko
2020-11-29 16:22                             ` Jean Louis
2020-11-26 18:07                         ` Dr. Arne Babenhauserheide
2020-11-26 23:09                       ` David Rogers
2020-11-27  0:43                         ` Tim Cross
2020-11-27  2:56                         ` Jean Louis
2020-11-23 16:07                   ` One vs many directories Texas Cyberthal
2020-11-23 19:20                     ` Jean Louis
2020-11-24  7:55                       ` Ihor Radchenko
2020-11-28 16:16                         ` Jean Louis
2020-11-28 16:33                           ` Christopher Dimech
2020-11-25  6:57                       ` Texas Cyberthal
2020-11-25  9:51                         ` Jean Louis
2020-11-25 10:39                           ` Texas Cyberthal
2020-11-25 11:02                             ` Jean Louis
2020-11-26 16:04                               ` Texas Cyberthal
2020-11-26 17:31                                 ` Jean Louis
2020-11-27  9:00                                   ` Texas Cyberthal
2020-11-27 10:45                                     ` Jean Louis
2020-11-28  8:18                                       ` Texas Cyberthal
2020-11-28 10:09                                         ` Jean Louis
2020-11-29  6:18                                           ` Texas Cyberthal
2020-11-29  6:53                                             ` Jean Louis
2020-11-30  7:35                                               ` Texas Cyberthal
2020-11-30  7:50                                                 ` Ihor Radchenko
2020-11-30 10:25                                                   ` Texas Cyberthal
2020-11-30 10:57                                                 ` Jean Louis
2020-11-30 12:27                                                   ` Ihor Radchenko
2020-11-30 12:28                                                   ` Ihor Radchenko
2020-11-30 19:00                                                     ` Jean Louis
2020-12-02  2:56                                                       ` Ihor Radchenko
2020-12-02  6:14                                                         ` Jean Louis
2020-12-02  7:23                                                           ` Ihor Radchenko
2020-11-21 16:55       ` Jean Louis
2020-11-21 22:48         ` Dr. Arne Babenhauserheide
2020-11-22  0:48           ` Jean Louis
2020-11-22  2:47             ` briangpowell
2020-11-22 17:55               ` Jean Louis
2020-11-21  6:12 ` Palak Mathur
2020-11-21  9:04   ` Jean Louis
2020-11-21  6:36 ` Jean Louis
2020-11-21  7:17   ` Texas Cyberthal
2020-11-21  9:53     ` Jean Louis
2020-11-21 10:15       ` Tim Cross
2020-11-21 11:18         ` Jean Louis
2020-11-21 14:44       ` Texas Cyberthal
2020-11-21 15:45         ` Jean Louis
2020-11-23  5:40     ` Ihor Radchenko
2020-11-24  9:00       ` Jean Louis
2020-11-24  9:45         ` Eric S Fraga
2020-11-24  9:51           ` Jean Louis
2020-11-24 11:42             ` Eric S Fraga
2020-11-24 13:13               ` Diego Zamboni
2020-11-24 13:49                 ` Jean Louis
2020-11-24 17:02                 ` Jean Louis
2020-11-24 18:50                   ` Dr. Arne Babenhauserheide
2020-11-24 18:58                     ` Jean Louis
2020-11-25  6:39                       ` Tim Cross
2020-11-25 12:38                         ` Local variables insecurities - " Jean Louis
2020-11-25 13:05                           ` Eric S Fraga
2020-11-25 13:13                             ` Jean Louis
2020-11-25 13:58                               ` Eric S Fraga
2020-11-25 14:07                                 ` Jean Louis
2020-11-25 20:54                                   ` Tim Cross
2020-11-25 22:09                                     ` Jean Louis
2020-11-26  2:06                                       ` Tom Gillespie
2020-11-26  5:06                                         ` Jean Louis
2020-11-26  5:31                                         ` Jean Louis
2020-11-26  6:18                                           ` Tom Gillespie
2020-11-26  9:10                                             ` Jean Louis
2020-11-26 11:44                                           ` Detlef Steuer
2020-11-26 12:06                                             ` Jean Louis
2020-11-26  5:34                                         ` Greg Minshall
2020-11-26  5:49                                           ` Jean Louis
2020-11-26  8:39                             ` Christian Moe
2020-11-25  8:10                       ` Dr. Arne Babenhauserheide
2020-11-25  8:36                         ` Local variables liberties Jean Louis
2020-11-24 20:11                     ` One vs many directories Tom Gillespie
2020-11-24 20:39                       ` Tim Cross
2020-11-25  4:54                         ` Jean Louis
2020-11-25  5:54                           ` Tim Cross
2020-11-25  7:01                             ` Local variables issue - " Jean Louis
2020-11-25  5:06                         ` Jean Louis
2020-11-25  7:00                           ` Tim Cross
2020-11-25  8:23                             ` Security issues in Emacs packages Jean Louis
2020-11-25  9:07                               ` tomas
2020-11-25  9:26                                 ` Jean Louis
2020-11-25 10:41                                   ` tomas
2020-11-25 22:46                               ` Tim Cross
2020-11-25 23:07                                 ` Jean Louis
2020-11-25 23:39                                   ` Tim Cross
2020-11-26  5:24                                     ` Jean Louis
2020-11-26  6:46                                       ` Tim Cross
2020-11-26  5:29                                 ` Greg Minshall
2020-11-26  5:53                                   ` Jean Louis
2020-11-26  6:35                                   ` Tim Cross
2020-11-26 12:27                                     ` Greg Minshall
2020-11-26 22:20                                       ` Tim Cross
2020-11-27  2:19                                         ` Jean Louis
2020-11-27  4:42                                         ` Greg Minshall
2020-11-25  4:44                       ` One vs many directories Jean Louis
2020-11-25 10:19           ` org-sbe to automate some source block executions Jean Louis
2020-11-25 11:39             ` Ihor Radchenko
2020-11-25 15:06               ` Jean Louis
2020-11-25 11:46           ` One vs many directories Jean Louis
2020-11-25 13:07             ` Eric S Fraga
2020-11-25 13:14               ` Jean Louis
2020-11-25 13:12             ` Ihor Radchenko
2020-11-25 13:32               ` Jean Louis
2020-11-24 18:47         ` Dr. Arne Babenhauserheide
2020-11-24 18:54           ` Jean Louis
2020-11-25  8:14             ` Dr. Arne Babenhauserheide
2020-11-25  8:46               ` Jean Louis
2020-11-25 11:46                 ` Ihor Radchenko
2020-11-26 12:47                   ` Jean Louis
2020-11-26 13:27                     ` Ihor Radchenko
2020-12-02 10:12                       ` Jean Louis
2020-12-02  9:49                   ` Jean Louis
2020-11-26  3:47           ` Ihor Radchenko
2020-11-26  3:32         ` Ihor Radchenko
2020-11-26 11:58           ` Jean Louis
2020-11-29  7:56             ` Ihor Radchenko
2020-11-29 17:57               ` Jean Louis
2020-11-21 13:41 ` Jonathan McHugh
2020-11-21 14:04   ` Jean Louis

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://orgmode.org

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAMUm490kEZh5GqLxwNqog1pM0OqUG3_bYoLqQnM9v+db=_my8A@mail.gmail.com' \
    --to=texas.cyberthal@gmail.com \
    --cc=arne_bab@web.de \
    --cc=emacs-orgmode@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Org-mode mailing list

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://orgmode.org/list/0 list/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 list list/ https://orgmode.org/list \
		emacs-orgmode@gnu.org
	public-inbox-index list

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.emacs.orgmode
	nntp://news.gmane.io/gmane.emacs.orgmode


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git