Shell Code Blocks in Babel
Introduction
A shell is a user interface for interacting with system services. File management, process execution, and operating system monitoring can all be done with a shell. Many shells are plain text whereas some support graphics or are themselves fully graphical. Shells often provide a programming language as well as access to system utilities, such as the GNU Core Utilities1. Users can stitch utilities together and create their own. Similarities exist between shells and, though standards exist (such as POSIX), there is no guarantee that what works in one shell will work in another2. Each shell is a separate application.
Org Babel works with several text based shells.
Run commands in separate shells…3
#+begin_src sh :results output echo PID: "$$" #+end_src #+RESULTS: : PID: 19056 #+begin_src sh :results output echo PID: "$$" #+end_src #+RESULTS: : PID: 19059
…as part of a single instance…
#+begin_src sh :results output :session shared echo PID: "$$" X=1 #+end_src #+RESULTS: : PID: 19066 #+begin_src sh :results output :session shared echo PID: "$$" echo X was set to "$X" #+end_src #+RESULTS: : PID: 19066 : X was set to 1
…asynchronously…
#+begin_src sh :session execute-asynchronously :async echo "Execute session blocks in the background" sleep 3 echo "Using the :async header" #+end_src #+RESULTS: : Execute session blocks in the background : Using the :async header
…or as a standalone shell command.
#+title: shell command (shebang) example This is a literate script file. The script we'll make can be used to ask someone what operating system they're using. First, we define a reference. #+name: their-os Linux The reference is then passed as stdin into a script which gets executed as a command according to the shebang (within a shell specified by the block language). We get a different response depending on who we ask. Let's test it by asking RMS: #+begin_src bash :results output :shebang #!/usr/bin/env bash :stdin their-os :cmdline RMS :tangle ask_for_os.sh # call as ./ask_for_os.sh NAME, where NAME is who to ask if [ -z "$1" ]; then asked="$USER" else asked="$1" fi echo Hi, "$asked"! What operating system are you using? read my_os if [ "$asked" = "RMS" ]; then echo You\'re using GNU/"$my_os"! elif [ "$asked" = "Linus" ]; then echo You\'re using "$my_os"! else echo You\'re using `uname -o`! fi #+end_src #+RESULTS: : Hi, RMS! What operating system are you using? : You're using GNU/Linux!
Shell blocks can accept args, work with stdin, be tangled, use noweb, and more4.
Some commands try to read standard input if it is available.
In interactive sessions it may be hidden due to time interval between typed
commands, but be careful when they are used in scripts Org source blocks.
BASH FAQ #89
warns concerning ssh
and ffmpeg
. Either explicitly specify
</dev/null
as input or use here-document syntax.
ssh example.org 'sed -i -e s/foo/bar/ file.txt' </dev/null ssh example.org 'tee >>file.log' <<"EOF" Added by org-babel EOF
(Quotes around "EOF" delimiter suppress variable expansion in the text.)
Tools may have dedicated options,
for example ssh -n
is a more concise way to avoid the pitfall.
Requirements and Setup
Org Babel can run many different shells such as sh, bash, zsh, fish, csh, ash, dash, ksh, mksh, and posh.
- The shell must be present on the system and be accessable from the $PATH.
The Org Babel language facility must be set to load shell functionality5:
;; active Babel languages (org-babel-do-load-languages 'org-babel-load-languages '((shell . t)))
TIP: All that's needed is
(shell . t)
, regardless of the intended shell (e.g. dash, fish, etc.).6NOTE: The
ob-shell.el
module used to be namedob-sh.el
. This was changed in Org 8.27.The desired shell should be given in place of the <lang> header arg.
For example, to run using dash8:
#+begin_src dash :result output if [ `basename $SHELL` == bash ]; then echo bash else echo dash fi #+end_src #+RESULTS: : dash
Header Arguments
Shell blocks support many common headers, as well as several headers specific only to shells.
Common:
Shell specific:
Results
:results {output, value}
output
returns stdout, defaultvalue
returns exit code
Supported types: table, list, and file.
Examples:
Define the following 2D-array:
#+name: make_array #+begin_src bash declare -a array m=4 n=3 for ((i=0; i<m; i++)) do for ((j=0; j<n; j++)) do a[${i},${j}]=$RANDOM done done for ((i=0; i<m; i++)) do for ((j=0; j<n; j++)) do echo -ne "${a[${i},${j}]}\t" done echo done #+end_src
When called with :results output table
or :results table
#+RESULTS: | 19323 | 14951 | 2805 | | 19323 | 14951 | 2805 | | 19323 | 14951 | 2805 | | 19323 | 14951 | 2805 |
When called with :results output list
or :results list
#+RESULTS: - (29607 15726 14035) - (29607 15726 14035) - (29607 15726 14035) - (29607 15726 14035)
When called with :results file :file my_output.txt
, a file named "my_output.txt" is
created which contains the output. The results contains a link to the file.
#+RESULTS: file:my_output.txt #+begin_src sh :results output cat my_output.txt #+end_src #+RESULTS: : 1526 5064 6484 : 1526 5064 6484 : 1526 5064 6484 : 1526 5064 6484
Dir
:dir <path>
Use the :dir
header to specify the default directory to use during
execution.
TIP Use :dir
to execute commands on a remote server. See Choosing
a working directory to learn more about :dir
.
Example:
Open an SSH connection to a server and create a file if it doesn't already exist.
#+begin_src bash :results output :dir /ssh:user@localhost:/home/user if [ ! -e "foo_file" ]; then echo "foo" > foo_file echo "Created foo_file" else echo "foo_file already exists!" fi #+end_src #+RESULTS: : Created foo_file
Example (with sessions):
It also works with sessions!
#+begin_src bash :results output :dir /ssh:user@localhost:/home/user :session *remote* if [ ! -e "foo_file" ]; then echo "foo" > foo_file echo "Created foo_file" else echo "foo_file already exists!" fi #+end_src #+RESULTS: : foo_file already exists!
Sessions
:session <session-name>
Use the header :session <session-name>
to run different code blocks
in the same shell instance. If <session-name>
is "none", blank, or
the :session
header arg is left out altogether, then the code block
evaluates in a temporary shell instance. Otherwise, a comint
(command-line interpreter) buffer with <session-name>
is created and
reused. The code block is then sent for evaluation and the results
inserted into the Org buffer. Blocks are processed synchronously
(i.e. they freeze Emacs until done).
Examples:
The following creates a session called "*my-session*
" and defines a
variable9. The next block uses the same shell instance and has
access to the variable from the previous block. Finally, a block with
another session is given. It uses a different shell instance and,
therefore, has no knowledge of the variable from the other session.
#+begin_src sh :results none :session *my-session* X=1 #+end_src #+begin_src sh :results output :session *my-session* echo X was set to "$X" #+end_src #+RESULTS: : X was set to 1 #+begin_src sh :results output :session *another-session* echo X was set to "$X" #+end_src #+RESULTS: : X was set to
Sessions are useful for isolating processes and for incremental development.
Async
:async <[yes|no]>
Run session blocks asynchronously with the :async
header. This
means Emacs will not freeze while waiting for the block to execute.
The :async
header accepts optional arguments of "yes" and "no".
When "yes", blocks are executed in a background process. When "no",
the block behaves like :session
(Emacs freezes until execution
completes). A "yes" argument is assumed unless the argument is "no".
NOTE: :async
requires the :session
header!
Examples:
A universally unique identifier (UUID) appears while an async block executes:
#+begin_src sh :results output :session test :async yes echo "Hello, world!" sleep 3 echo "Good-bye, cruel World..." #+end_src #+RESULTS: : d3aec7bd-c2a0-41f4-bd4d-edad7143cbbc
When the process completes, the UUID is replaced with the results:
#+begin_src sh :results output :session test :async yes echo "Hello, world!" sleep 3 echo "Good-bye, cruel World..." #+end_src #+RESULTS: : Hello, world! : Good-bye, cruel World...
Disable asynchronous execution by passing "no" as an argument:
#+begin_src sh :results output :session test :async no echo "Changing the :async argument to 'no'" sleep 3 echo "will run the block synchronously (freezing Emacs" echo "until the block completes)." #+end_src #+RESULTS: : Changing the :async argument to 'no' : will run the block synchronously (freezing Emacs : until the block completes).
Variables
:var <name_1>=<value_1> ... [<name_n>=<value_n>]
Use the :var
header to define variables in the shell environment.
Variables are defined using the following forms. Separate multiple variables with a space.
type | form |
---|---|
scalar | <name>=<value> |
1D-array* | <name>='(element1 element2 ...) |
*
Only works with Bash.
Example (scalar):
#+begin_src dash :var by_two=0 x=2 if [ "$by_two" = "0" ]; then echo $(($x * 2)) else echo $(($x * 3)) fi #+end_src #+RESULTS: : 4
Example (1D-array):
#+begin_src bash :results output :var arr='("apple" "banana" "cherry") echo The first element is... echo \"${arr[1]}\" #+end_src #+RESULTS: : The first element is... : "banana"
The :var
header lets results be passed between blocks, which allows blocks to act
like functions.
#+name: multiply_by_2 #+begin_src bash :var data="" :results output echo $(($data * 2)) #+end_src #+begin_src bash :post multiply_by_2(data=*this*) echo 3 #+end_src #+results: : 6
Standard Input
:stdin <element-name>
Use the :stdin
header arg to pass named Org elements, such as code block results,
to a shell process as standard input.
NOTE When :stdin
is used, the block is evaluated in a temporary shell, regardless
of whether a :session
is declared.
Example:
#+name: my-org-element This is something referenced as stdin. The stdin can even be multiple lines! #+begin_src sh :stdin my-org-element :results output cat #+end_src #+RESULTS: : This is something referenced as stdin. : The stdin can even be multiple lines!
Example:
#+name: my-input 3 #+begin_src bash :stdin my-input :results output read n echo You entered: "$n" #+end_src #+RESULTS: : You entered: 3
Example:
#+name: my-input-block #+begin_src bash echo "$(uname -o)" x=$(ps -p $PPID -o comm=) echo "${x:1:5}" #+end_src #+RESULTS: my-input-block | GNU/Linux | | emacs | #+begin_src bash :stdin my-input-block echo $(cut -f 1 -d "/") rocks! #+end_src #+RESULTS: : GNU emacs rocks!
Command-line Arguments
:cmdline <arg_1> ... [arg_n]
Use the :cmdline
header arg to pass arguments to a shell command.
The arguments are passed verbatim as
<system shell> -c <temporary file with src block contents> <arguments>
:cmdline
is used verbatim and is interpreted by shell. Org mode
does not prevent shell expansion/interpretation. This, for example,
means that
#+begin_src sh :cmdline arg ; touch /tmp/not-an-arg ... #+end_src
will evaluate touch
command as well.
When :cmdline
is used, the block is evaluated in a temporary shell,
regardless of whether a :session
is declared.
#+begin_src sh :cmdline arg1 arg2 echo "$1" echo "$2" #+end_src #+RESULTS: | arg1 | | arg2 |
Shebang (standalone scripts)
:shebang [shebang]
Use the :shebang
header arg to process a block as a command10.
Pass an interpreter directive (e.g. #!/bin/bash
) to override the declared shell. If no shebang
is provided, the block language is used (e.g. #+begin_src bash
).
NOTE When the :shebang
header arg is used, the block is evaluated in a temporary
shell, regardless of whether a :session
is declared.
TIP The :shebang
header arg is useful for when a script spans several
lines. Without the :shebang
header arg, block lines are sent to the shell process
in sequence which may result in unwanted output characters.
Example (remove unwanted characters):
#+begin_src sh :results output :session unwanted-chars X=1 echo Setting X... sleep 3 echo X was set to "$X" #+end_src #+RESULTS: : Setting X... : sh-5.1$ X was set to 1 #+begin_src sh :results output :shebang X=1 echo Setting X... sleep 3 echo X was set to "$X" #+end_src #+RESULTS: : Setting X... : X was set to 1
Example (override):
#+begin_src bash :shebang #!/usr/bin/env dash if [ `basename $SHELL` == bash ]; then echo bash; else echo dash; fi #+end_src #+RESULTS: : dash
Footnotes:
Tangle means to extract various blocks into a single source file. Noweb is a syntax used to insert/reference other code blocks. See the following for more details:
The symbol "shell" loads functionality for all supported
shells. Specifically, the car of a org-babel-load-languages
element
is used to load the corresponding module. The car of (shell . t)
is
"shell" which is used to load ob-shell.el
, the Org Babel module that
handles shell interactions. There is not a separate module for each
shell.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/etc/ORG-NEWS#n3995
* Version 8.2 ** Incompatible changes *** =ob-sh.el= renamed to =ob-shell= This may require two changes in user config. 1. In =org-babel-do-load-languages=, change =(sh . t)= to =(shell . t)=. 2. Edit =local.mk= files to change the value of =BTEST_OB_LANGUAGES= to remove "sh" and include "shell".
This is a weird example. A better one might simply check the $SHELL environment variable. The reason the example doesn't is that the shell process created by a source block is a subprocess of Emacs. As such, it retains some environment variables, namely $SHELL. This document was written on a system using bash as it's main shell. This means that although dash is used to create the subprocess, the $SHELL variable still points to bash. To be more precise, the example uses the fact that "==" is valid bash syntax, but invalid in dash. The first condition is met on bash, echoing "bash", whereas it fails on dash, echoing "dash". This way, we can be sure that the block is really using dash.
Using asterisks, or "earmuffs", for the session name is not mandatory.
In some other systems there is a convention of choosing variable names that begin and end with ‘*’. We don’t use that convention in Emacs Lisp, so please don’t use it in your programs. (Emacs uses such names only for special-purpose buffers.)
https://www.gnu.org/software/emacs/manual/html_node/elisp/Coding-Conventions.html