Shell Code Blocks in Babel

Babel shell support, including: sh, bash, zsh, fish, csh, ash, dash, ksh, mksh, and posh.

Introduction

A shell is a user interface for interacting with system services. File management, process execution, and operating system monitoring can all be done with a shell. Many shells are plain text whereas some support graphics or are themselves fully graphical. Shells often provide a programming language as well as access to system utilities, such as the GNU Core Utilities¹. Users can stitch utilities together and create their own. Similarities exist between shells and, though standards exist (such as POSIX), there is no guarantee that what works in one shell will work in another². Each shell is a separate application.

Org Babel works with several text based shells.

Run commands in separate shells…³

#+begin_src sh :results output
  echo PID: "$$"
#+end_src

#+RESULTS:
: PID: 19056

#+begin_src sh :results output
  echo PID: "$$"
#+end_src

#+RESULTS:
: PID: 19059

…as part of a single instance…

#+begin_src sh :results output :session shared
  echo PID: "$$"
  X=1
#+end_src

#+RESULTS:
: PID: 19066

#+begin_src sh :results output :session shared
  echo PID: "$$"
  echo X was set to "$X"
#+end_src

#+RESULTS:
: PID: 19066
: X was set to 1

…asynchronously…

#+begin_src sh :session execute-asynchronously :async
echo "Execute session blocks in the background"
sleep 3
echo "Using the :async header"
#+end_src

#+RESULTS:
: Execute session blocks in the background
: Using the :async header

…or as a standalone shell command.

#+title: shell command (shebang) example

This is a literate script file.  The script we'll make can be used to ask
someone what operating system they're using.  First, we define a reference.

#+name: their-os
Linux

The reference is then passed as stdin into a script which gets executed as a
command according to the shebang (within a shell specified by the block
language).  We get a different response depending on who we ask.

Let's test it by asking RMS:

#+begin_src bash :results output :shebang #!/usr/bin/env bash :stdin their-os :cmdline RMS :tangle ask_for_os.sh

  # call as ./ask_for_os.sh NAME, where NAME is who to ask

  if [ -z "$1" ]; then
      asked="$USER"
  else
      asked="$1"
  fi

  echo Hi, "$asked"! What operating system are you using?
  read my_os

  if [ "$asked" = "RMS" ]; then
      echo You\'re using GNU/"$my_os"!
  elif [ "$asked" = "Linus" ]; then
      echo You\'re using "$my_os"!
  else
      echo You\'re using `uname -o`!
  fi
#+end_src

#+RESULTS:
: Hi, RMS! What operating system are you using?
: You're using GNU/Linux!

Shell blocks can accept args, work with stdin, be tangled, use noweb, and more⁴.

Requirements and Setup

Org Babel can run many different shells such as sh, bash, zsh, fish, csh, ash, dash, ksh, mksh, and posh.

The shell must be present on the system and be accessable from the $PATH.
The Org Babel language facility must be set to load shell functionality⁵:
```
;; active Babel languages
(org-babel-do-load-languages
'org-babel-load-languages
'((shell . t)))
```
TIP: All that's needed is (shell . t), regardless of the intended shell (e.g. dash, fish, etc.).⁶

NOTE: The ob-shell.el module used to be named ob-sh.el. This was changed in Org 8.2⁷.

The desired shell should be given in place of the <lang> header arg.

For example, to run using dash⁸:

#+begin_src dash :result output
  if [ `basename $SHELL` == bash ]; then
    echo bash
  else
    echo dash
  fi
#+end_src

#+RESULTS:
: dash

Header Arguments

Shell blocks support many common headers, as well as several headers specific only to shells.

Common:

:results
:dir
:session
- :async
:var
:noweb
:tangle

Shell specific:

:stdin
:cmdline
:shebang

Results

:results {output, value}

output returns stdout, default
value returns exit code

Supported types: table, list, and file.

Examples:

Define the following 2D-array:

#+name: make_array
#+begin_src bash
declare -a array

m=4
n=3
for ((i=0; i<m; i++))
do
    for ((j=0; j<n; j++))
    do
        a[${i},${j}]=$RANDOM
    done
done
for ((i=0; i<m; i++))
do
    for ((j=0; j<n; j++))
    do
        echo -ne "${a[${i},${j}]}\t"
    done
    echo
done
#+end_src

When called with :results output table or :results table

#+RESULTS:
| 19323 | 14951 | 2805 |
| 19323 | 14951 | 2805 |
| 19323 | 14951 | 2805 |
| 19323 | 14951 | 2805 |

When called with :results output list or :results list

#+RESULTS:
- (29607 15726 14035)
- (29607 15726 14035)
- (29607 15726 14035)
- (29607 15726 14035)

When called with :results file :file my_output.txt, a file named "my_output.txt" is created which contains the output. The results contains a link to the file.

#+RESULTS:
file:my_output.txt

#+begin_src sh :results output
  cat my_output.txt
#+end_src

#+RESULTS:
: 1526	5064	6484
: 1526	5064	6484
: 1526	5064	6484
: 1526	5064	6484

Dir

:dir <path>

Use the :dir header to specify the default directory to use during execution.

TIP Use :dir to execute commands on a remote server. See Choosing a working directory to learn more about :dir.

Example:

Open an SSH connection to a server and create a file if it doesn't already exist.

#+begin_src bash :results output :dir /ssh:user@localhost:/home/user
if [ ! -e  "foo_file" ];
then
    echo "foo" > foo_file
    echo "Created foo_file"
else
    echo "foo_file already exists!"
fi
#+end_src

#+RESULTS:
: Created foo_file

Example (with sessions):

It also works with sessions!

#+begin_src bash :results output :dir /ssh:user@localhost:/home/user :session *remote*
if [ ! -e  "foo_file" ];
then
    echo "foo" > foo_file
    echo "Created foo_file"
else
    echo "foo_file already exists!"
fi
#+end_src

#+RESULTS:
: foo_file already exists!

Sessions

:session <session-name>

Use the header :session <session-name> to run different code blocks in the same shell instance. If <session-name> is "none", blank, or the :session header arg is left out altogether, then the code block evaluates in a temporary shell instance. Otherwise, a comint (command-line interpreter) buffer with <session-name> is created and reused. The code block is then sent for evaluation and the results inserted into the Org buffer. Blocks are processed synchronously (i.e. they freeze Emacs until done).

Examples:

The following creates a session called "*my-session*" and defines a variable⁹. The next block uses the same shell instance and has access to the variable from the previous block. Finally, a block with another session is given. It uses a different shell instance and, therefore, has no knowledge of the variable from the other session.

#+begin_src sh :results none :session *my-session*
  X=1
#+end_src

#+begin_src sh :results output :session *my-session*
  echo X was set to "$X"
#+end_src

#+RESULTS:
: X was set to 1

#+begin_src sh :results output :session *another-session*
  echo X was set to "$X"
#+end_src

#+RESULTS:
: X was set to

Sessions are useful for isolating processes and for incremental development.

Async

:async <[yes|no]>

Run session blocks asynchronously with the :async header. This means Emacs will not freeze while waiting for the block to execute.

The :async header accepts optional arguments of "yes" and "no". When "yes", blocks are executed in a background process. When "no", the block behaves like :session (Emacs freezes until execution completes). A "yes" argument is assumed unless the argument is "no".

NOTE: :async requires the :session header!

Examples:

A universally unique identifier (UUID) appears while an async block executes:

#+begin_src sh :results output :session test :async yes
echo "Hello, world!"
sleep 3
echo "Good-bye, cruel World..."
#+end_src

#+RESULTS:
: d3aec7bd-c2a0-41f4-bd4d-edad7143cbbc

When the process completes, the UUID is replaced with the results:

#+begin_src sh :results output :session test :async yes
echo "Hello, world!"
sleep 3
echo "Good-bye, cruel World..."
#+end_src

#+RESULTS:
: Hello, world!
: Good-bye, cruel World...

Disable asynchronous execution by passing "no" as an argument:

#+begin_src sh :results output :session test :async no
echo "Changing the :async argument to 'no'"
sleep 3
echo "will run the block synchronously (freezing Emacs"
echo "until the block completes)."
#+end_src

#+RESULTS:
: Changing the :async argument to 'no'
: will run the block synchronously (freezing Emacs
: until the block completes).

Variables

:var <name_1>=<value_1> ... [<name_n>=<value_n>]

Use the :var header to define variables in the shell environment.

Variables are defined using the following forms. Separate multiple variables with a space.

type	form
scalar	`<name>=<value>`
1D-array*	`<name>='(element1 element2 ...)`

* Only works with Bash.

Example (scalar):

#+begin_src dash :var by_two=0  x=2
  if [ "$by_two" = "0" ]; then
    echo $(($x * 2))
  else
    echo $(($x * 3))
  fi
#+end_src

#+RESULTS:
: 4

Example (1D-array):

#+begin_src bash :results output :var arr='("apple" "banana" "cherry")
  echo The first element is...
  echo \"${arr[1]}\"
#+end_src

#+RESULTS:
: The first element is...
: "banana"

The :var header lets results be passed between blocks, which allows blocks to act like functions.

#+name: multiply_by_2
#+begin_src bash :var data="" :results output
  echo $(($data * 2))
#+end_src

#+begin_src bash :post multiply_by_2(data=*this*)
  echo 3
#+end_src

#+results:
: 6

Standard Input

:stdin <element-name>

Use the :stdin header arg to pass named Org elements, such as code block results, to a shell process as standard input.

NOTE When :stdin is used, the block is evaluated in a temporary shell, regardless of whether a :session is declared.

Example:

#+name: my-org-element
This is something referenced as stdin.
The stdin can even be multiple lines!

#+begin_src sh :stdin my-org-element :results output
cat
#+end_src

#+RESULTS:
: This is something referenced as stdin.
: The stdin can even be multiple lines!

Example:

#+name: my-input
3

#+begin_src bash :stdin my-input :results output
read n
echo You entered: "$n"
#+end_src

#+RESULTS:
: You entered: 3

Example:

#+name: my-input-block
#+begin_src bash
  echo "$(uname -o)"
  x=$(ps -p $PPID -o comm=)
  echo "${x:1:5}"
#+end_src

#+RESULTS: my-input-block
| GNU/Linux |
| emacs     |

#+begin_src bash :stdin my-input-block
  echo $(cut -f 1 -d "/") rocks!
#+end_src

#+RESULTS:
: GNU emacs rocks!

Command-line Arguments

:cmdline <arg_1> ... [arg_n]

Use the :cmdline header arg to pass arguments to a shell command.

NOTE When :cmdline is used, the block is evaluated in a temporary shell, regardless of whether a :session is declared.

#+begin_src sh :cmdline arg1 arg2
  echo "$1"
  echo "$2"
#+end_src

#+RESULTS:
| arg1 |
| arg2 |

Shebang (standalone scripts)

:shebang [shebang]

Use the :shebang header arg to process a block as a command¹⁰.

Pass an interpreter directive (e.g. #!/bin/bash) to override the declared shell. If no shebang is provided, the block language is used (e.g. #+begin_src bash).

NOTE When the :shebang header arg is used, the block is evaluated in a temporary shell, regardless of whether a :session is declared.

TIP The :shebang header arg is useful for when a script spans several lines. Without the :shebang header arg, block lines are sent to the shell process in sequence which may result in unwanted output characters.

Example (remove unwanted characters):

#+begin_src sh :results output :session unwanted-chars
  X=1
  echo Setting X...
  sleep 3
  echo X was set to "$X"
#+end_src

#+RESULTS:
: Setting X...
: sh-5.1$ X was set to 1

#+begin_src sh :results output :shebang
  X=1
  echo Setting X...
  sleep 3
  echo X was set to "$X"
#+end_src

#+RESULTS:
: Setting X...
: X was set to 1

Example (override):

#+begin_src bash :shebang #!/usr/bin/env dash
  if [ `basename $SHELL` == bash ]; then
    echo bash;
  else
    echo dash;
  fi
#+end_src

#+RESULTS:
: dash

Footnotes:

https://en.wikipedia.org/wiki/List_of_GNU_Core_Utilities_commands

https://opensource.com/article/19/7/what-posix-richard-stallman-explains

https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#index-_0024_0024

⁴

Tangle means to extract various blocks into a single source file. Noweb is a syntax used to insert/reference other code blocks. See the following for more details:

⁵

https://orgmode.org/worg/org-contrib/babel/languages/index.html#configure

⁶

The symbol "shell" loads functionality for all supported shells. Specifically, the car of a org-babel-load-languages element is used to load the corresponding module. The car of (shell . t) is "shell" which is used to load ob-shell.el, the Org Babel module that handles shell interactions. There is not a separate module for each shell.

⁷

https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/etc/ORG-NEWS#n3995

* Version 8.2

** Incompatible changes
*** =ob-sh.el= renamed to =ob-shell=
This may require two changes in user config.

1. In =org-babel-do-load-languages=, change =(sh . t)= to =(shell . t)=.
2. Edit =local.mk= files to change the value of =BTEST_OB_LANGUAGES=
   to remove "sh" and include "shell".

⁸

This is a weird example. A better one might simply check the $SHELL environment variable. The reason the example doesn't is that the shell process created by a source block is a subprocess of Emacs. As such, it retains some environment variables, namely $SHELL. This document was written on a system using bash as it's main shell. This means that although dash is used to create the subprocess, the $SHELL variable still points to bash. To be more precise, the example uses the fact that "==" is valid bash syntax, but invalid in dash. The first condition is met on bash, echoing "bash", whereas it fails on dash, echoing "dash". This way, we can be sure that the block is really using dash.

⁹

Using asterisks, or "earmuffs", for the session name is not mandatory.

In some other systems there is a convention of choosing variable names that begin and end with ‘*’. We don’t use that convention in Emacs Lisp, so please don’t use it in your programs. (Emacs uses such names only for special-purpose buffers.)

https://www.gnu.org/software/emacs/manual/html_node/elisp/Coding-Conventions.html

¹⁰

https://en.wikipedia.org/wiki/Shebang_(Unix)