Google Calendar Synchronization

To get a real synchronization between org-mode and Google Calendar you need to sync two ways. We cover one way of handling the synchronization in the next two sections and also mention some other ways of synchronization at the end.

Google Calendar offers access to each calendar via a hidden/secret url. That is, a url that only you know about and is very hard to guess for other people. You can use this to download your calendar in an iCalendar (.ics) format, which we then can rewrite into something usable for org-mode. For this conversion there luckily already exists a script written by Eric S. Fraga¹. This is the latest version:

#!/usr/bin/awk -f
# awk script for converting an iCal formatted file to a sequence of org-mode headings.
# this may not work in general but seems to work for day and timed events from Google's
# calendar, which is really all I need right now...
#
# usage:
#   awk -f THISFILE < icalinputfile.ics > orgmodeentries.org
#
# Note: change org meta information generated below for author and
# email entries!
#
# Known bugs:
# - not so much a bug as a possible assumption: date entries with no time
#   specified are assumed to be independent of the time zone.
#
# Eric S Fraga
# 20100629 - initial version
# 20100708 - added end times to timed events
#          - adjust times according to time zone information
#          - fixed incorrect transfer for entries with ":" embedded within the text
#          - added support for multi-line summary entries (which become headlines)
# 20100709 - incorporated time zone identification
#          - fixed processing of continuation lines as Google seems to
#            have changed, in the last day, the number of spaces at
#            the start of the line for each continuation...
#          - remove backslashes used to protect commas in iCal text entries
# no further revision log after this as the file was moved into a git
# repository...
#
# Last change: 2011.01.28 16:08:03
#----------------------------------------------------------------------------------

# a function to take the iCal formatted date+time, convert it into an
# internal form (seconds since time 0), and adjust according to the
# local time zone (specified by +-UTC_offset calculated in the BEGIN
# section)

function datetimestamp(input)
{
    # convert the iCal Date+Time entry to a format that mktime can understand

    # datespec in UTC, i.e. ending with Z
    UTC = "no"
    UTC = gensub("([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])T([0-9][0-9])([0-9][0-9])([0-9][0-9])Z.*[\r]*", "yes", "g", input);

    # parse date and time
    datespec  = gensub("([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])T([0-9][0-9])([0-9][0-9])([0-9][0-9]).*[\r]*", "\\1 \\2 \\3 \\4 \\5 \\6", "g", input);

    # print "date spec : " datespec; convert this date+time into
    # seconds from the beginning of time and include adjustment for
    # time zone, as determined in the BEGIN section below.  For time
    # zone adjustment, I have not tested edge effects, specifically
    # what happens when UTC time is a different day to local time and
    # especially when an event with a duration crosses midnight in UTC
    # time.  It should work but...
    if(UTC == "yes")
        timestamp = mktime(datespec) + UTC_offset ;
    else
        timestamp = mktime(datespec);

    # print "adjusted    : " timestamp
    # print "Time stamp  : " strftime("%Y-%m-%d %H:%M", timestamp);
    return timestamp;
}

BEGIN {
    ### config section
    max_age =  7; # in days
                  # set this to -1 to get all entries or to N>0 to only get 
                  # that start or end less than N days ago
    ### end config section

    # use a colon to separate the type of data line from the actual contents
    FS = ":";

    # determine the number of seconds to use for adjusting for time
    # zone difference from UTC.  This is used in the function
    # datetimestamp above.  The time zone information returned by
    # strftime() is in hours * 100 so we multiply by 36 to get
    # seconds.  This does not work for time zones that are not an
    # integral multiple of hours (e.g. Newfoundland)
    UTC_offset = gensub("([+-])0", "\\1", "", strftime("%z")) * 36;

    date = "";
    entry = ""
    first = 1;                  # true until an event has been found
    headline = ""
    icalentry = ""  # the full entry for inspection
    id = ""
    indescription = 0;
    lasttimestamp=-1;

    print "#+TITLE:     Main Google calendar entries"
    print "#+AUTHOR:    Eric S Fraga"
    print "#+EMAIL:     e.fraga@ucl.ac.uk"
    print "#+DESCRIPTION: converted using the ical2org awk script"
    print "#+CATEGORY: google"
    print "#+STARTUP: hidestars"
    print "#+STARTUP: overview"
    print " "
}

# continuation lines (at least from Google) start with two spaces
# if the continuation is after a description or a summary, append the entry
# to the respective variable

/^[ ]+/ { 
    if (indescription) {
        entry = entry gensub("\r", "", "g", gensub("^[ ]+", "", "", $0));
    } else if (insummary) {
        summary = summary gensub("\r", "", "g", gensub("^[ ]+", "", "", $0))
    }
    icalentry = icalentry "\n" $0
}

/^BEGIN:VEVENT/ {
    # start of an event.  if this is the first, output the preamble from the iCal file
    if (first) {
        print "* COMMENT original iCal preamble"
        print gensub("\r", "", "g", icalentry)
        icalentry = ""
    }
    first = false;
}
# any line that starts at the left with a non-space character is a new data field

/^[A-Z]/ {
    # we ignore DTSTAMP lines as they change every time you download
    # the iCal format file which leads to a change in the converted
    # org file as I output the original input.  This change, which is
    # really content free, makes a revision control system update the
    # repository and confuses.
    if (! index("DTSTAMP", $1)) icalentry = icalentry "\n" $0
    # this line terminates the collection of description and summary entries
    indescription = 0;
    insummary = 0;
}

# this type of entry represents a day entry, not timed, with date stamp YYYYMMDD

/^DTSTART;VALUE=DATE/ {
    datetmp = gensub("([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])(.*[\r])", "\\1T000000\\2", "g", $2)
    date = strftime("%Y-%m-%d %a %H:%M", datetimestamp(datetmp));
    if(max_age>0)     lasttimestamp = datetimestamp(datetmp);
}
/^DTEND;VALUE=DATE/ {
    datetmp = gensub("([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])(.*[\r])", "\\1T000000\\2", "g", $2)
    time2 = strftime("%Y-%m-%d %a %H:%M", datetimestamp(datetmp));
    date = date ">--<" time2;
    if(max_age>0)     lasttimestamp = datetimestamp(datetmp);
}

# this represents a timed entry with date and time stamp YYYYMMDDTHHMMSS
# we ignore the seconds

/^DTSTART[:;][^V]/ {
    date = strftime("%Y-%m-%d %a %H:%M", datetimestamp($2));
    if(max_age>0)     lasttimestamp = datetimestamp($2);
    # print date;
}

# and the same for the end date; here we extract only the time and append this to the 
# date+time found by the DTSTART entry.  We assume that entry was there, of course.
# should probably add some error checking here!  In time...

/^DTEND[:;][^V]/ {
    # print $0
    time2 = strftime("%Y-%m-%d %a %H:%M", datetimestamp($2));
    date = date ">--<" time2;
    if(max_age>0)     lasttimestamp = datetimestamp($2);
}

# The description will the contents of the entry in org-mode.
# this line may be continued.

/^DESCRIPTION/ { 
    $1 = "";
    entry = entry gensub("\r", "", "g", $0);
    indescription = 1;
}

# the summary will be the org heading

/^SUMMARY/ { 
    $1 = "";
    summary = gensub("\r", "", "g", $0);
    insummary = 1;
}

# the unique ID will be stored as a property of the entry

/^UID/ { 
    id = gensub("\r", "", "g", $2);
}

/^LOCATION/ {
    location = gensub("\r", "", "g", $2);
}

/^STATUS/ {
    status = gensub("\r", "", "g", $2);
}

# when we reach the end of the event line, we output everything we
# have collected so far, creating a top level org headline with the
# date/time stamp, unique ID property and the contents, if any

/^END:VEVENT/ {
    #output event
    if(max_age<0 || ( lasttimestamp>0 && systime()<lasttimestamp+max_age*24*60*60 ) )
    {
    # translate \n sequences to actual newlines and unprotect commas (,)
    print "* " gensub("\\\\,", ",", "g", gensub("\\\\n", " ", "g", summary))
    print "  <" date ">"
    print "  :PROPERTIES:"
    print "  :ID:       " id
    if(length(location))
        print "  :LOCATION: " location
    if(length(status))
        print "  :STATUS: " status
    print "  :END:"
    # for the entry, convert all embedded "\n" strings to actual newlines
    print ""
    # translate \n sequences to actual newlines and unprotect commas (,)
    if(length(entry)>1)
    print gensub("\\\\,", ",", "g", gensub("\\\\n", "\n", "g", entry));
    print "** COMMENT original iCal entry"
    print gensub("\r", "", "g", icalentry)
    }
    summary = ""
    date = ""
    location = ""
    status = ""
    entry = ""
    icalentry = ""
    indescription = 0
    insummary = 0
    lasttimestamp = -1
}

# Local Variables:
# time-stamp-line-limit: 1000
# time-stamp-format: "%04y.%02m.%02d %02H:%02M:%02S"
# time-stamp-active: t
# time-stamp-start: "Last change:[ \t]+"
# time-stamp-end: "$"
# End:

With this you can test your Google Calendar to org-mode synchronization by following these steps:

1. Download the above script and save it as 'ical2org'. Make sure that the script is in your PATH and don't forget to set the executable flag (chmod u+x ical2org). You can also customize the script a bit by changing the variables in the config section of the script.
2. Find your private URL for your calendar
   • Log into Google Calendar
   • Goto Settings
   • Click on the calendar you want to export to org-mode
   • At the bottom of the page find the 'private address' section and your ical link Use the 'reset private urls' if you need to, that is if you don't see a unique url.
3. Download the url This can be done for example using 'wget <url>'
4. Transform into org-file Use the downloaded script via 'ical2org < icsfile > orgfile'. Where icsfile is the path to the file you downloaded from Google and orgfile is the org-mode file you want to create.
5. Add the orgfile to your agenda and test

If this all works you can automate the task via cron. Create a script such as the following that will automatically download and convert your calendar. Make sure that this file is only readable by you (chmod 700 <file>), since it will contain the url to your Google calendar.

#!/bin/bash

# customize these
WGET=<path to wget>
ICS2ORG=<path to ical2org>
ICSFILE=<path for icsfile>
ORGFILE=<path to orgfile>
URL=<url to your private Google calendar>

# no customization needed below

$WGET -O $ICSFILE $URL
$ICS2ORG < $ICSFILE > $ORGFILE

automate this via cron by adding something like the following to your crontab:

5,20,35,50 * * * * <path to above script> &> /dev/null #sync my org files

This will sync every 15 minutes starting at 5 minutes past the hour.

There are at least two possible paths to get the information into Google:

1. export from org mode to .ics; upload .ics to a public web server giving it a hidden/secret name; tell Google to import this .ics file
2. use googlecl to import event when you create them into Google calendar (update entries won't be reflected in Google). See ²

The first one has the disadvantage that the item won't show up in your "main" calendar and therefore you can't easily share them with others. Nevertheless, this route is relatively easy and therefore we'll discuss it below. The second option (as described in the link) should work well, if you don't need to change things.

Also keep in mind that your mileage will vary, since everything described on this page works for some people, but perhaps not for you… if this is the case, feel free to ask on the org-email list and perhaps we can add missing features.

Back to the topic. To implement 1., we need org to export an .ics file, which can be achieved using the function: org-export-icalendar-combine-agenda-files. This will export all entries in you agenda. If you only want to export certain ones, you can set up a filter. For this we will define a filter function and then tell Emacs to use this filter. The filter shown here, will exclude items with a category "google" (for example from the ical2org script) and "private" and also only export entries that have a date and a time range set (that is, a start and a end time stamp). You can modify the function though to do anything you want!

;;; define categories that should be excluded
(setq org-export-exclude-category (list "google" "private"))

;;; define filter. The filter is called on each entry in the agenda.
;;; It defines a regexp to search for two timestamps, gets the start
;;; and end point of the entry and does a regexp search. It also
;;; checks if the category of the entry is in an exclude list and
;;; returns either t or nil to skip or include the entry.

(defun org-mycal-export-limit ()
  "Limit the export to items that have a date, time and a range. Also exclude certain categories."
  (setq org-tst-regexp "<\\([0-9]\\{4\\}-[0-9]\\{2\\}-[0-9]\\{2\\} ... [0-9]\\{2\\}:[0-9]\\{2\\}[^\r\n>]*?\
\)>")
  (setq org-tstr-regexp (concat org-tst-regexp "--?-?" org-tst-regexp))
  (save-excursion
    ; get categories
    (setq mycategory (org-get-category))
    ; get start and end of tree
    (org-back-to-heading t)
    (setq mystart    (point))
    (org-end-of-subtree)
    (setq myend      (point))
    (goto-char mystart)
    ; search for timerange
    (setq myresult (re-search-forward org-tstr-regexp myend t))
    ; search for categories to exclude
    (setq mycatp (member mycategory org-export-exclude-category))
    ; return t if ok, nil when not ok
    (if (and myresult (not mycatp)) t nil)))

;;; activate filter and call export function
(defun org-mycal-export () 
  (let ((org-icalendar-verify-function 'org-mycal-export-limit))
    (org-export-icalendar-combine-agenda-files)))

To use these function you can include the above code in your .emacs file and then in case you run Emacs server call

emacsclient -e "(save-excursion (org-mycal-export))"

in your shell to generate the .ics file.

If you want to export also TODO items that have a SCHEDULED timestamp, you can set

(setq org-icalendar-use-scheduled '(todo-start event-if-todo))

in your .emacs.

Another Emacs variable that you might want to look into is: org-icalendar-honor-noexport-tag.

You can now automate this via a cron job to generate updated .ics files.

The next step is to give the file a cryptic name (so that other people have a hard time accessing your file, also make sure that your web server doesn't show an index for your directory, etc.) and copy it to a public accessible web server. Then log into your Google calendar and in the left column under "Other calendars" use "Add"->"Add by url" to point Google at your freshly generated .ics file and you should be all set up. Once you done this Google will every now and then (every few hours?) look for a newer version of your .ics file and include this in your calendar.