emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* A requires/provides approach to linking source code blocks
@ 2021-07-08 16:24 Marko Schuetz-Schmuck
  2021-07-08 19:32 ` Tim Cross
  2021-07-09  7:29 ` Stefan Nobis
  0 siblings, 2 replies; 12+ messages in thread
From: Marko Schuetz-Schmuck @ 2021-07-08 16:24 UTC (permalink / raw)
  To: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 843 bytes --]

Dear All,

AFAIU in the current support for literate programming I can establish
sequence between blocks by either tangling the entire file whereby the
blocks are written to the source code file in the sequence in which they
appear in the org-mode file or I can name the blocks and use noweb
linking to explicitly state the precise sequence in which the blocks
appear in the source code file.

I would find it useful to have a more declarative way for specifying
sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
capture dependency and then have the exporter compute a sequence
satisfying these. I would think that this makes the maintenance of the
dependencies more convenient.

I'd be interested in feedback on the idea.

Please include my email address in the reply as I am not subscribed to
the list.

Best regards,

Marko

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 223 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: A requires/provides approach to linking source code blocks
  2021-07-08 16:24 A requires/provides approach to linking source code blocks Marko Schuetz-Schmuck
@ 2021-07-08 19:32 ` Tim Cross
  2021-07-09  8:22   ` Slowing down new features Timothy
                     ` (2 more replies)
  2021-07-09  7:29 ` Stefan Nobis
  1 sibling, 3 replies; 12+ messages in thread
From: Tim Cross @ 2021-07-08 19:32 UTC (permalink / raw)
  To: Marko Schuetz-Schmuck; +Cc: emacs-orgmode


Marko Schuetz-Schmuck <MarkoSchuetz@web.de> writes:

> [[PGP Signed Part:Undecided]]
> Dear All,
>
> AFAIU in the current support for literate programming I can establish
> sequence between blocks by either tangling the entire file whereby the
> blocks are written to the source code file in the sequence in which they
> appear in the org-mode file or I can name the blocks and use noweb
> linking to explicitly state the precise sequence in which the blocks
> appear in the source code file.
>
> I would find it useful to have a more declarative way for specifying
> sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
> capture dependency and then have the exporter compute a sequence
> satisfying these. I would think that this makes the maintenance of the
> dependencies more convenient.
>
> I'd be interested in feedback on the idea.
>
> Please include my email address in the reply as I am not subscribed to
> the list.
>

My concern here is with the additional complexity. This is already a
somewhat complex aspect of org mode and the behaviour you describe can
effectively be done using noweb, although as you say, not as
declarative in style.

I'm not sure the added complexity is worth the additional maintenance
overhead and I would be concerned regarding how much confusion this
could cause i.e. someone uses both noweb and declarative
provide/require. These additions are also likely to hamper any efforts
to improve performance - a topic which comes up on a fairly regular
basis. 

This could just be me, but recently, I'm becoming very concerned about
the growth of additional features and options in org mode. Many of the
new things being requested are for relatively narrow use cases and
represent functionality which can largely be achieved with existing
features. At the same time, there also seems to be a growing
frequency in patches which introduce regression bugs and patches to fix
patches.

This is all beginning to feel like we are running very close to the
tipping point at which time we will have something that is so complex
that only a very few people are able to maintain the code base and keep
the system stable. New maintainers are discouraged because of the code
complexity. We could end up in a position where really important issues
cannot be addressed or addressed in a timely manner because of the
overall complexity of the code base and time it takes to fix and test
and dependence on a very few number of maintainers who are already
swamped with work. 

At some point, I think we may want to consider a temporary freeze on new
functionality and spend a few months just focusing on bug fixes and code
base improvements or re-factoring. It might also be worthwhile providing
some guidelines or criteria/procedures for assessing proposed new
features to avoid a perception of new features being accepted/rejected
based on personality, loudness of voice or some other real or perceived
and irrelevant basis.


-- 
Tim Cross


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: A requires/provides approach to linking source code blocks
  2021-07-08 16:24 A requires/provides approach to linking source code blocks Marko Schuetz-Schmuck
  2021-07-08 19:32 ` Tim Cross
@ 2021-07-09  7:29 ` Stefan Nobis
  1 sibling, 0 replies; 12+ messages in thread
From: Stefan Nobis @ 2021-07-09  7:29 UTC (permalink / raw)
  To: Marko Schuetz-Schmuck; +Cc: emacs-orgmode

Marko Schuetz-Schmuck <MarkoSchuetz@web.de> writes:

> I would find it useful to have a more declarative way for specifying
> sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
> capture dependency and then have the exporter compute a sequence
> satisfying these.

I would say that declaring an explicit ordering of blocks via noweb is
quite declarative.

The tensions seems more to be between implicit and explicit ordering.
I'm quite a fan of letting the computer compute things, but in the
context of literate programming I tend to prefer explicit over
implicit. IMHO that's the main point of literate programming: Be more
explicit and document all the details. Try not to hide important
aspects.

If you have so many blocks that maintaining the order of the
concatenation seems like a burden and tiresome I would argue that you
are doing it wrong. Because in the end you need quite a good
understanding of the final order of the blocks or else maintaining the
implicit order via require/provide will also get out of hand.

I also second Tims concerns about the additional complexity and
feature creep.

-- 
Until the next mail...,
Stefan.


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Slowing down new features
  2021-07-08 19:32 ` Tim Cross
@ 2021-07-09  8:22   ` Timothy
  2021-07-09 13:38   ` A requires/provides approach to linking source code blocks Maxim Nikulin
  2021-07-09 20:27   ` Berry, Charles via General discussions about Org-mode.
  2 siblings, 0 replies; 12+ messages in thread
From: Timothy @ 2021-07-09  8:22 UTC (permalink / raw)
  To: Tim Cross; +Cc: emacs-orgmode, Marko Schuetz-Schmuck

[-- Attachment #1: Type: text/plain, Size: 2302 bytes --]

Hi Tim,

> This is all beginning to feel like we are running very close to the
> tipping point at which time we will have something that is so complex
> that only a very few people are able to maintain the code base and keep
> the system stable. New maintainers are discouraged because of the code
> complexity. We could end up in a position where really important issues
> cannot be addressed or addressed in a timely manner because of the
> overall complexity of the code base and time it takes to fix and test
> and dependence on a very few number of maintainers who are already
> swamped with work.
>
> At some point, I think we may want to consider a temporary freeze on new
> functionality and spend a few months just focusing on bug fixes and code
> base improvements or re-factoring. It might also be worthwhile providing
> some guidelines or criteria/procedures for assessing proposed new
> features to avoid a perception of new features being accepted/rejected
> based on personality, loudness of voice or some other real or perceived
> and irrelevant basis.

I’ve got some changes that improve existing features that I know I’d like to
push (and would be happy to maintain), but I get the same feeling as you that
overall that we’ve been working on a lot of new things and it would be worth
taking some time to refine what’s here instead of adding more.

That said, I’m not sure a hard freeze would be the best way to approach this,
I’d be more of a fan of just a shift in focus and maybe deferring some ideas to
be considered later.

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

FWIW, these are the new things I’d like to push:
• ob-julia
• inline src block fortification
• smart generated export preamble
• a new LaTeX src block syntax highlighting option (this actually isn’t that
  much code)

These are all currently maturing in my config, and in particular the “smart
generated export preamble” and new LaTeX syntax highlighting option have been
massaged into a state I’m very happy with over a number of months.

All the best,
*Timothy*

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: A requires/provides approach to linking source code blocks
  2021-07-08 19:32 ` Tim Cross
  2021-07-09  8:22   ` Slowing down new features Timothy
@ 2021-07-09 13:38   ` Maxim Nikulin
  2021-07-09 20:27   ` Berry, Charles via General discussions about Org-mode.
  2 siblings, 0 replies; 12+ messages in thread
From: Maxim Nikulin @ 2021-07-09 13:38 UTC (permalink / raw)
  To: emacs-orgmode

On 09/07/2021 02:32, Tim Cross wrote:
> Marko Schuetz-Schmuck writes:
> 
>> I would find it useful to have a more declarative way for specifying
>> sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
>> capture dependency and then have the exporter compute a sequence
>> satisfying these. I would think that this makes the maintenance of the
>> dependencies more convenient.
> 
> My concern here is with the additional complexity. This is already a
> somewhat complex aspect of org mode and the behaviour you describe can
> effectively be done using noweb, although as you say, not as
> declarative in style.

There was a feature request for dependencies between code blocks to run 
them in proper order a half of a year ago.

I think, it is better to try such feature as an extension of org, as a 
separate package. I suspect that some non-obvious issue may arise. 
Likely, to be convenient, it will be desired to autofill 
requires/provides attributes using some tools related to particular 
language.

The question is whether org code is organized in such way, so extensions 
of this kind are possible without code duplication. Maybe API could be 
adjusted. On the other hand, even maintaining of stable semi-internal 
API sometimes is significant burden.

P.S. I am reading the mail list through NNTP news.gmain.io. I have 
noticed that personal copies sometimes arrives without mail list 
address. Sorry if you will have to manually add it to reply.



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: A requires/provides approach to linking source code blocks
@ 2021-07-09 17:00 autofrettage
  2021-07-12  9:55 ` Alexander Adolf
  0 siblings, 1 reply; 12+ messages in thread
From: autofrettage @ 2021-07-09 17:00 UTC (permalink / raw)
  To: emacs-orgmode@gnu.org

Tim wrote:
> This could just be me, but recently, I'm becoming very concerned
> about the growth of additional features and options in org mode.

Count me in. I have been mostly been hanging around in the shadows, but this is serious enough for me to wave a flag on the right side.

I would go as far as saying that several suggestions have been so niche, as to be labeled feature bloats. They can be made available as user-added extensions through melpa, but should stay outside org itself.

just my ¢2

Rasmus


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: A requires/provides approach to linking source code blocks
  2021-07-08 19:32 ` Tim Cross
  2021-07-09  8:22   ` Slowing down new features Timothy
  2021-07-09 13:38   ` A requires/provides approach to linking source code blocks Maxim Nikulin
@ 2021-07-09 20:27   ` Berry, Charles via General discussions about Org-mode.
  2 siblings, 0 replies; 12+ messages in thread
From: Berry, Charles via General discussions about Org-mode. @ 2021-07-09 20:27 UTC (permalink / raw)
  To: Tim Cross; +Cc: emacs-orgmode@gnu.org, Marko Schuetz-Schmuck

Tim,

> On Jul 8, 2021, at 12:32 PM, Tim Cross <theophilusx@gmail.com> wrote:
> 
> My concern here is with the additional complexity. This is already a
> somewhat complex aspect of org mode and the behaviour you describe can
> effectively be done using noweb, although as you say, not as
> declarative in style.


This (and the rest of what you said) is very well taken.

I share your concern that adding features to an already rich feature suite will make babel very difficult to penetrate.

Best,
Chuck

p.s. Extracting code from src blocks need not depend on `org-babel-tangle' and friends. Custom exporters have to potential to render code using attribute keys and other export features.  And such exporters have the virtue of not adding complexity to the org code base.


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: A requires/provides approach to linking source code blocks
  2021-07-09 17:00 autofrettage
@ 2021-07-12  9:55 ` Alexander Adolf
  2021-07-13 19:18   ` Tom Gillespie
  0 siblings, 1 reply; 12+ messages in thread
From: Alexander Adolf @ 2021-07-12  9:55 UTC (permalink / raw)
  To: autofrettage; +Cc: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]


> On 9. Jul 2021, at 19:01, autofrettage <autofrettage@protonmail.ch> wrote:
> 
> Tim wrote:
>> This could just be me, but recently, I'm becoming very concerned
>> about the growth of additional features and options in org mode.
> 
> Count me in. I have been mostly been hanging around in the shadows, but this is serious enough for me to wave a flag on the right side.
> […]

Same concern here. 

  — Alex

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1944 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: A requires/provides approach to linking source code blocks
  2021-07-12  9:55 ` Alexander Adolf
@ 2021-07-13 19:18   ` Tom Gillespie
  2021-07-13 19:42     ` George Mauer
  2021-07-14  0:44     ` Tim Cross
  0 siblings, 2 replies; 12+ messages in thread
From: Tom Gillespie @ 2021-07-13 19:18 UTC (permalink / raw)
  To: emacs-orgmode; +Cc: Tim Cross

We have been receiving many new feature suggestions and requests
coming in for org babel. I think that Tim's suggestion is the right
one. Nearly all of these need to be implemented as an extension first
and tested independently. Further, even if this is done, it should be
clear that there is zero expectation that such extensions will be
incorporated.

Once I wrap up the formal grammar for org, one of the next things I
plan to work on is a clear specification for org babel. This is
critical because so many of the suggestions that come in deal with
individuals' specific problems and thus fail to account for how such
features interact with existing features and how the newly proposed
feature would block some other features in the future, confuse users,
etc. Such suggestions also often fail to account for increased
complexity, nor have they been exposed to a sufficient number of
examples to reveal fundamental ambiguities in how they could be
interpreted. The issues with variable behavior between ob langs for
:pre :post :prologue :epilogue etc. are already enough to keep us busy
for quite some time.

With regard to this thread in particular, it is of some interest, but
there are fundamental issues, including the fact that certain
languages (e.g. racket) expect module code to exist somewhere on the
file system. There are ways around many of these issues, in fact there
are likely many ways around any individual issue, so org babel needs
to systematically consider the issues and provide a clear
specification, or at least a guide for how such cases should be
handled.

To give an example from one of my continual pain points: I start
writing python or racket in an org src block and then I want it to be
a library so that it can be reused by other code both inside and
outside the org file without having to resort to noweb.

What is the best way to handle this? I don't know. Right now I tangle
things and resort to awful hacks for the reuse-in-this-org-file case, but
I'm guessing there is a better generic solution which would allow _any_
org block to be exported as a library instead of nowebbed in.

Before jumping for any particular suggestion for how to handle this
we need to explore the diversity of cases that various ob langs
present, so that we can find a solution that will work for all of
them. After all, packaging code to a library for reuse is an
extremely common pattern that org babel should be able to
abstract, but it is a major undertaking, not just the addition of a
keyword here and there.

In short I suggest that we issue a general moratorium on new org babel
feature suggestions until we can stabilize what we already have and
provide a clear specification for correct behavior. Until we have that spec
we could encourage users to create extensions that implement those
features.

Best,
Tom


PS The other next thing that I am working on might be another way out
for this particular feature request. Namely, it is simplifying and
extending org keyword syntax so that new keywords (with options) and
associated keywords can be specified using keyword syntax within a
single org file. This would make it possible to get useful high level
keyword behavior in a single file without burdening the core
implementation with more special cases for associated keywords, and it
would allow users to write small elisp functions that could do some of
what is suggested here, all without need to add anything to the core
org implementation.


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: A requires/provides approach to linking source code blocks
  2021-07-13 19:18   ` Tom Gillespie
@ 2021-07-13 19:42     ` George Mauer
  2021-07-14  0:44     ` Tim Cross
  1 sibling, 0 replies; 12+ messages in thread
From: George Mauer @ 2021-07-13 19:42 UTC (permalink / raw)
  To: emacs-orgmode; +Cc: Tom Gillespie

[-- Attachment #1: Type: text/plain, Size: 4848 bytes --]

In implementing an extension you might consider doing it as a generalized
form of what I did with ob-racket (https://github.com/togakangaroo/ob-racket
).

I think it is best to rely on the import/require/include mechanism of the
language you're using. Pretty much all of them support adjacent files. The
problem with code block execution is that it expands the block and writes
it to a temporary directory in a non-predictable location. If you want to
reference other blocks, you need them to *also* write into files in the
same folder.

What I do in ob-racket is support a header :adjacent-file which allows you
to pass in the name of another block and then writes it into the same
folder as the file to be executed with a predictable name (the name of the
block)

So something like this

  #+begin_src racket :adjacent-file stacker-reader-expander.rkt
    #lang reader "./stacker-reader-expander.rkt"

Will make sure the block named `stacker-reader-expander.rkt` is written
into that same temporary directory with that same name. There's a few rough
edges but for the most part it works well. Would love to see someone take
this and generalize it into its own extension.

On Tue, Jul 13, 2021 at 2:18 PM Tom Gillespie <tgbugs@gmail.com> wrote:

> We have been receiving many new feature suggestions and requests
> coming in for org babel. I think that Tim's suggestion is the right
> one. Nearly all of these need to be implemented as an extension first
> and tested independently. Further, even if this is done, it should be
> clear that there is zero expectation that such extensions will be
> incorporated.
>
> Once I wrap up the formal grammar for org, one of the next things I
> plan to work on is a clear specification for org babel. This is
> critical because so many of the suggestions that come in deal with
> individuals' specific problems and thus fail to account for how such
> features interact with existing features and how the newly proposed
> feature would block some other features in the future, confuse users,
> etc. Such suggestions also often fail to account for increased
> complexity, nor have they been exposed to a sufficient number of
> examples to reveal fundamental ambiguities in how they could be
> interpreted. The issues with variable behavior between ob langs for
> :pre :post :prologue :epilogue etc. are already enough to keep us busy
> for quite some time.
>
> With regard to this thread in particular, it is of some interest, but
> there are fundamental issues, including the fact that certain
> languages (e.g. racket) expect module code to exist somewhere on the
> file system. There are ways around many of these issues, in fact there
> are likely many ways around any individual issue, so org babel needs
> to systematically consider the issues and provide a clear
> specification, or at least a guide for how such cases should be
> handled.
>
> To give an example from one of my continual pain points: I start
> writing python or racket in an org src block and then I want it to be
> a library so that it can be reused by other code both inside and
> outside the org file without having to resort to noweb.
>
> What is the best way to handle this? I don't know. Right now I tangle
> things and resort to awful hacks for the reuse-in-this-org-file case, but
> I'm guessing there is a better generic solution which would allow _any_
> org block to be exported as a library instead of nowebbed in.
>
> Before jumping for any particular suggestion for how to handle this
> we need to explore the diversity of cases that various ob langs
> present, so that we can find a solution that will work for all of
> them. After all, packaging code to a library for reuse is an
> extremely common pattern that org babel should be able to
> abstract, but it is a major undertaking, not just the addition of a
> keyword here and there.
>
> In short I suggest that we issue a general moratorium on new org babel
> feature suggestions until we can stabilize what we already have and
> provide a clear specification for correct behavior. Until we have that spec
> we could encourage users to create extensions that implement those
> features.
>
> Best,
> Tom
>
>
> PS The other next thing that I am working on might be another way out
> for this particular feature request. Namely, it is simplifying and
> extending org keyword syntax so that new keywords (with options) and
> associated keywords can be specified using keyword syntax within a
> single org file. This would make it possible to get useful high level
> keyword behavior in a single file without burdening the core
> implementation with more special cases for associated keywords, and it
> would allow users to write small elisp functions that could do some of
> what is suggested here, all without need to add anything to the core
> org implementation.
>
>

[-- Attachment #2: Type: text/html, Size: 5553 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: A requires/provides approach to linking source code blocks
  2021-07-13 19:18   ` Tom Gillespie
  2021-07-13 19:42     ` George Mauer
@ 2021-07-14  0:44     ` Tim Cross
  2021-07-14  1:53       ` Samuel Wales
  1 sibling, 1 reply; 12+ messages in thread
From: Tim Cross @ 2021-07-14  0:44 UTC (permalink / raw)
  To: Tom Gillespie; +Cc: emacs-orgmode


Tom Gillespie <tgbugs@gmail.com> writes:

[snip]

>
> Once I wrap up the formal grammar for org, one of the next things I
> plan to work on is a clear specification for org babel. This is
> critical because so many of the suggestions that come in deal with
> individuals' specific problems and thus fail to account for how such
> features interact with existing features and how the newly proposed
> feature would block some other features in the future, confuse users,
> etc. Such suggestions also often fail to account for increased
> complexity, nor have they been exposed to a sufficient number of
> examples to reveal fundamental ambiguities in how they could be
> interpreted. The issues with variable behavior between ob langs for
> :pre :post :prologue :epilogue etc. are already enough to keep us busy
> for quite some time.
>

Yes, that clearly summs up my concerns as well. Often, a 'fix' for some
problem can seem straight-forward, possibly even trivial, when
considered only within one person's use case/workflow. Part of what
makes org so powerful is the level of flexibility it supports. However,
this flexibility makes it extremely difficult to consider, or even know
about, all the existing use cases. 

I suspect that once we have some formal specifications for babel, the
next step will be to develop some good unit tests to verify these
specifications. This would at least highlight/alert developers to
unforeseen impact from changes and alert them to things they may
not have considered. 

> With regard to this thread in particular, it is of some interest, but
> there are fundamental issues, including the fact that certain
> languages (e.g. racket) expect module code to exist somewhere on the
> file system. There are ways around many of these issues, in fact there
> are likely many ways around any individual issue, so org babel needs
> to systematically consider the issues and provide a clear
> specification, or at least a guide for how such cases should be
> handled.
>
> To give an example from one of my continual pain points: I start
> writing python or racket in an org src block and then I want it to be
> a library so that it can be reused by other code both inside and
> outside the org file without having to resort to noweb.
>

This is an interesting point. I think a number of languages have
challenges here. I run into very similar issues with Clojure.

To some extent, I think this is a grey area within the literate
programming paradigm. The original literate programming model was
developed at a time when most languages were compiled rather then
interpreted. You generated source code, compiled it and then ran it.

These days, many 'modern' languages are based around an interpreter and
concepts like 'just in time' compilation. In org mode, things become
even more complex because in addition to generation (tangling) of code,
we also want to have evaluation of code blocks, plus we have added the
concept of 'sessions'.

> What is the best way to handle this? I don't know. Right now I tangle
> things and resort to awful hacks for the reuse-in-this-org-file case, but
> I'm guessing there is a better generic solution which would allow _any_
> org block to be exported as a library instead of nowebbed in.
>
> Before jumping for any particular suggestion for how to handle this
> we need to explore the diversity of cases that various ob langs
> present, so that we can find a solution that will work for all of
> them. After all, packaging code to a library for reuse is an
> extremely common pattern that org babel should be able to
> abstract, but it is a major undertaking, not just the addition of a
> keyword here and there.
>

Agree. I'm not convinced we really understand the requirements here and
more exploration and specification is required. The more we add 'simple'
extensions, options, keywords etc the more likely it is we will make
this a much harder process and will likely result in even larger
'breakage' once we do define a clearer specification.

> In short I suggest that we issue a general moratorium on new org babel
> feature suggestions until we can stabilize what we already have and
> provide a clear specification for correct behavior. Until we have that spec
> we could encourage users to create extensions that implement those
> features.
>

Yes. While it might sound harsh or overly limiting, I do think such a
moratorium may be required. We may be able to lift it once we have some
core specifications in place. We would still accept bug fixes (though we
may need to provide some clarity on what is a bug fix and what is a
feature enhancement/change - I regularly see posts flagged as bug fixes
which are actually feature enhancements or extensions). 
>
> PS The other next thing that I am working on might be another way out
> for this particular feature request. Namely, it is simplifying and
> extending org keyword syntax so that new keywords (with options) and
> associated keywords can be specified using keyword syntax within a
> single org file. This would make it possible to get useful high level
> keyword behavior in a single file without burdening the core
> implementation with more special cases for associated keywords, and it
> would allow users to write small elisp functions that could do some of
> what is suggested here, all without need to add anything to the core
> org implementation.

Sounds interesting. To some extent, recognising there may be at least
two types of code blocks (library code, executable/driver code) and at
least two types of languages (compiled/interpreted) may help identify
base code block permutation and required options, keywords etc. However,
we are unlikely to find a definitive set which supports all use
cases/workflows and the ability to easily extend/customise behaviours
would be very useful. 
-- 
Tim Cross


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: A requires/provides approach to linking source code blocks
  2021-07-14  0:44     ` Tim Cross
@ 2021-07-14  1:53       ` Samuel Wales
  0 siblings, 0 replies; 12+ messages in thread
From: Samuel Wales @ 2021-07-14  1:53 UTC (permalink / raw)
  To: Tim Cross; +Cc: Tom Gillespie, emacs-orgmode

one thing about org that i think has been making it complex, in
addition to number of features, is non-orthogonality.

On 7/13/21, Tim Cross <theophilusx@gmail.com> wrote:
>
> Tom Gillespie <tgbugs@gmail.com> writes:
>
> [snip]
>
>>
>> Once I wrap up the formal grammar for org, one of the next things I
>> plan to work on is a clear specification for org babel. This is
>> critical because so many of the suggestions that come in deal with
>> individuals' specific problems and thus fail to account for how such
>> features interact with existing features and how the newly proposed
>> feature would block some other features in the future, confuse users,
>> etc. Such suggestions also often fail to account for increased
>> complexity, nor have they been exposed to a sufficient number of
>> examples to reveal fundamental ambiguities in how they could be
>> interpreted. The issues with variable behavior between ob langs for
>> :pre :post :prologue :epilogue etc. are already enough to keep us busy
>> for quite some time.
>>
>
> Yes, that clearly summs up my concerns as well. Often, a 'fix' for some
> problem can seem straight-forward, possibly even trivial, when
> considered only within one person's use case/workflow. Part of what
> makes org so powerful is the level of flexibility it supports. However,
> this flexibility makes it extremely difficult to consider, or even know
> about, all the existing use cases.
>
> I suspect that once we have some formal specifications for babel, the
> next step will be to develop some good unit tests to verify these
> specifications. This would at least highlight/alert developers to
> unforeseen impact from changes and alert them to things they may
> not have considered.
>
>> With regard to this thread in particular, it is of some interest, but
>> there are fundamental issues, including the fact that certain
>> languages (e.g. racket) expect module code to exist somewhere on the
>> file system. There are ways around many of these issues, in fact there
>> are likely many ways around any individual issue, so org babel needs
>> to systematically consider the issues and provide a clear
>> specification, or at least a guide for how such cases should be
>> handled.
>>
>> To give an example from one of my continual pain points: I start
>> writing python or racket in an org src block and then I want it to be
>> a library so that it can be reused by other code both inside and
>> outside the org file without having to resort to noweb.
>>
>
> This is an interesting point. I think a number of languages have
> challenges here. I run into very similar issues with Clojure.
>
> To some extent, I think this is a grey area within the literate
> programming paradigm. The original literate programming model was
> developed at a time when most languages were compiled rather then
> interpreted. You generated source code, compiled it and then ran it.
>
> These days, many 'modern' languages are based around an interpreter and
> concepts like 'just in time' compilation. In org mode, things become
> even more complex because in addition to generation (tangling) of code,
> we also want to have evaluation of code blocks, plus we have added the
> concept of 'sessions'.
>
>> What is the best way to handle this? I don't know. Right now I tangle
>> things and resort to awful hacks for the reuse-in-this-org-file case, but
>> I'm guessing there is a better generic solution which would allow _any_
>> org block to be exported as a library instead of nowebbed in.
>>
>> Before jumping for any particular suggestion for how to handle this
>> we need to explore the diversity of cases that various ob langs
>> present, so that we can find a solution that will work for all of
>> them. After all, packaging code to a library for reuse is an
>> extremely common pattern that org babel should be able to
>> abstract, but it is a major undertaking, not just the addition of a
>> keyword here and there.
>>
>
> Agree. I'm not convinced we really understand the requirements here and
> more exploration and specification is required. The more we add 'simple'
> extensions, options, keywords etc the more likely it is we will make
> this a much harder process and will likely result in even larger
> 'breakage' once we do define a clearer specification.
>
>> In short I suggest that we issue a general moratorium on new org babel
>> feature suggestions until we can stabilize what we already have and
>> provide a clear specification for correct behavior. Until we have that
>> spec
>> we could encourage users to create extensions that implement those
>> features.
>>
>
> Yes. While it might sound harsh or overly limiting, I do think such a
> moratorium may be required. We may be able to lift it once we have some
> core specifications in place. We would still accept bug fixes (though we
> may need to provide some clarity on what is a bug fix and what is a
> feature enhancement/change - I regularly see posts flagged as bug fixes
> which are actually feature enhancements or extensions).
>>
>> PS The other next thing that I am working on might be another way out
>> for this particular feature request. Namely, it is simplifying and
>> extending org keyword syntax so that new keywords (with options) and
>> associated keywords can be specified using keyword syntax within a
>> single org file. This would make it possible to get useful high level
>> keyword behavior in a single file without burdening the core
>> implementation with more special cases for associated keywords, and it
>> would allow users to write small elisp functions that could do some of
>> what is suggested here, all without need to add anything to the core
>> org implementation.
>
> Sounds interesting. To some extent, recognising there may be at least
> two types of code blocks (library code, executable/driver code) and at
> least two types of languages (compiled/interpreted) may help identify
> base code block permutation and required options, keywords etc. However,
> we are unlikely to find a definitive set which supports all use
> cases/workflows and the ability to easily extend/customise behaviours
> would be very useful.
> --
> Tim Cross
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html


^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2021-07-14  1:53 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-08 16:24 A requires/provides approach to linking source code blocks Marko Schuetz-Schmuck
2021-07-08 19:32 ` Tim Cross
2021-07-09  8:22   ` Slowing down new features Timothy
2021-07-09 13:38   ` A requires/provides approach to linking source code blocks Maxim Nikulin
2021-07-09 20:27   ` Berry, Charles via General discussions about Org-mode.
2021-07-09  7:29 ` Stefan Nobis
  -- strict thread matches above, loose matches on Subject: below --
2021-07-09 17:00 autofrettage
2021-07-12  9:55 ` Alexander Adolf
2021-07-13 19:18   ` Tom Gillespie
2021-07-13 19:42     ` George Mauer
2021-07-14  0:44     ` Tim Cross
2021-07-14  1:53       ` Samuel Wales

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).