Re: Google calendar to org mode script and a feature request for agenda

Re: Google calendar to org mode script and a feature request for agenda

[Torsten Wagner]

[-- Attachment #1: Type: text/plain, Size: 1953 bytes --]

Just by reading and without the possibility to test it. Why you multiply the hour with 36 instead of 3600 to get seconds?

Grettings
 Torsten 

Eric S Fraga <ucecesf@ucl.ac.uk> wrote:

>On Wed, 7 Jul 2010 17:38:45 -0300, Daniel Martins <danielemc@gmail.com> wrote:
>> 
>> Eric,
>> 
>> Your awk seems to get timed appts in GMT
>> 
>> How can I adapt it to GMT-3
>> 
>> Daniel
>
>Okay!  I think I've got this working for any time zone (as well as
>adding some more functionality -- read the prologue in the script for
>info).  
>
>Attached is the awk script.  I use this from within a shell script (on
>Linux) that essentially does this:
>
>--8<---------------cut here---------------start------------->8---
>ICS=basic.ics
>ORG=googlecalendar.org
>AWK=ical2org.awk
>
># get the Google calendar
>wget  http://www.google.com/calendar/ical/[...]/basic.ics
>
># convert the ical entries to org format, adjusting for the
># time zone information
>
># this next command yields hours from UTC, + or -, times 100
># Note: this does not cater for those people living in time zones
># that are not aligned with discrete hours (e.g. Newfoundland)...
>timezone=$(date +%z | sed 's/^\([+-]\)0/\1/')
>
># convert this to seconds for use in the awk script
>seconds=$(($timezone*36))
>
># and now process the ics file with appropriate time zone
>awk -f $AWK --assign SECONDS=$seconds < $ICS > $ORG
>--8<---------------cut here---------------end--------------->8---
>
>Please test this all out and let me know if it works.  If the date and
>sed commands work, you should be adjusting the times by -3*3600=-10800
>seconds.  This seems to be working for me with BST (aka GMT+1).
>
>eric
>
>-- 
>Eric S Fraga
>GnuPG: 8F5C 279D 3907 E14A 5C29  570D C891 93D8 FFFC F67D
>
>_______________________________________________
>Emacs-orgmode mailing list
>Please use `Reply All' to send replies to the list.
>Emacs-orgmode@gnu.org
>http://lists.gnu.org/mailman/listinfo/emacs-orgmode

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Google calendar to org mode script and a feature request for agenda

[Nick Dokos]

Torsten Wagner  <torsten.wagner@gmail.com> wrote:

> Just by reading and without the possibility to test it. Why you multiply the hour with 36 instead of 3600 to get seconds?
> 
> > ...
> ># this next command yields hours from UTC, + or -, times 100
                                                      ^^^^^^^^^
There's your factor of 100 (although I, too, have not tested it.)

> ># Note: this does not cater for those people living in time zones
> ># that are not aligned with discrete hours (e.g. Newfoundland)...
> >timezone=$(date +%z | sed 's/^\([+-]\)0/\1/')
> >

Cheers,
Nick

Re: Google calendar to org mode script and a feature request for agenda

[Eric S Fraga]

On Fri, 09 Jul 2010 09:57:48 +0900, Torsten Wagner <torsten.wagner@gmail.com> wrote:
> 
> Just by reading and without the possibility to test it. Why you multiply the hour with 36 instead of 3600 to get seconds?

Because the value returned by the %z format for the date command is in
"hundred hours" form so that, for instance, GMT+1 gives +0100...
Furthermore, note that I remove the leading 0 or else the bash
arithmetic evaluation treats the number as octal.

The comment is a little terse!  Sorry about that.
-- 
Eric S Fraga
GnuPG: 8F5C 279D 3907 E14A 5C29  570D C891 93D8 FFFC F67D

Re: [google] updated awk script for google to org conversion

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]

I've updated the awk script for converting Google calendar iCal export to org:

- the time zone calculation is now down completely within the awk
  script, not requiring any external calculations.

- fixed handling of continuation lines as Google seems to have
  changed, since last night, the format of the export iCal entries.

Again, comments, questions and criticisms all more than welcome!

Please note that the shell command required to use this is now much simpler:

  awk -f thisawkscript < icsfile > orgfile

as there is no longer any need to define the SECONDS variable.

Enjoy! :)

eric

[-- Attachment #2: ical2org.awk --]
[-- Type: text/plain, Size: 6961 bytes --]

# awk script for converting an iCal formatted file to a sequence of org-mode headings.
# this may not work in general but seems to work for day and timed events from Google's
# calendar, which is really all I need right now...
#
# usage:
#   awk -f THISFILE < icalinputfile.ics > orgmodeentries.org
#
# Note: change org meta information generated below for author and
# email entries!
#
# Known bugs:
# - not so much a bug as a possible assumption: date entries with no time
#   specified are assumed to be independent of the time zone.
#
# Eric S Fraga
# 20100629 - initial version
# 20100708 - added end times to timed events
#          - adjust times according to time zone information
#          - fixed incorrect transfer for entries with ":" embedded within the text
#          - added support for multi-line summary entries (which become headlines)
# 20100709 - incorporated time zone identification
#          - fixed processing of continuation lines as Google seems to
#            have changed, in the last day, the number of spaces at
#            the start of the line for each continuation...
#          - remove backslashes used to protect commas in iCal text entries
#
# Last change: 2010.07.09 10:42:54
#----------------------------------------------------------------------------------

# a function to take the iCal formatted date+time, convert it into an
# internal form (seconds since time 0), and adjust according to the
# local time zone (specified by +-seconds calculated in the BEGIN
# section)

function datetimestamp(input)
{
    # convert the iCal Date+Time entry to a format that mktime can understand
    datespec = gensub("([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])T([0-9][0-9])([0-9][0-9])([0-9][0-9]).*[\r]*", "\\1 \\2 \\3 \\4 \\5 \\6", "g", input);
    # print "date spec : " datespec; convert this date+time into
    # seconds from the beginning of time and include adjustment for
    # time zone, as determined in the BEGIN section below.  For time
    # zone adjustment, I have not tested edge effects, specifically
    # what happens when UTC time is a different day to local time and
    # especially when an event with a duration crosses midnight in UTC
    # time.  It should work but...
    timestamp = mktime(datespec) + seconds;
    # print "adjusted    : " timestamp
    # print "Time stamp  : " strftime("%Y-%m-%d %H:%M", timestamp);
    return timestamp;
}

BEGIN {
    # use a colon to separate the type of data line from the actual contents
    FS = ":";
    
    # determine the number of seconds to use for adjusting for time
    # zone difference from UTC.  This is used in the function
    # datetimestamp above.  The time zone information returned by
    # strftime() is in hours * 100 so we multiply by 36 to get
    # seconds.  This does not work for time zones that are not an
    # integral multiple of hours (e.g. Newfoundland)
    seconds = gensub("([+-])0", "\\1", "", strftime("%z")) * 36;
    
    date = "";
    entry = ""
    first = 1;			# true until an event has been found
    headline = ""
    icalentry = ""  # the full entry for inspection
    id = ""
    indescription = 0;
    time2given = 0;
    
    print "#+TITLE:     Main Google calendar entries"
    print "#+AUTHOR:    Eric S Fraga"
    print "#+EMAIL:     e.fraga@ucl.ac.uk"
    print "#+DESCRIPTION: converted using the ical2org awk script"
    print "#+CATEGORY: google"
    print " "
}

# continuation lines (at least from Google) start with two spaces
# if the continuation is after a description or a summary, append the entry
# to the respective variable

/^[ ]+/ { 
    if (indescription) {
	entry = entry gensub("\r", "", "g", gensub("^[ ]+", "", "", $0));
    } else if (insummary) {
	summary = summary gensub("\r", "", "g", gensub("^[ ]+", "", "", $0))
    }
    icalentry = icalentry "\n" $0
}

/^BEGIN:VEVENT/ {
    # start of an event.  if this is the first, output the preamble from the iCal file
    if (first) {
	print "* COMMENT original iCal preamble"
	print gensub("\r", "", "g", icalentry)
	icalentry = ""
    }
    first = false;
}
# any line that starts at the left with a non-space character is a new data field

/^[A-Z]/ {
    # we ignore DTSTAMP lines as they change every time you download
    # the iCal format file which leads to a change in the converted
    # org file as I output the original input.  This change, which is
    # really content free, makes a revision control system update the
    # repository and confuses.
    if (! index("DTSTAMP", $1)) icalentry = icalentry "\n" $0
    # this line terminates the collection of description and summary entries
    indescription = 0;
    insummary = 0;
}

# this type of entry represents a day entry, not timed, with date stamp YYYYMMDD

/^DTSTART;VALUE=DATE/ {
    date = gensub("([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9]).*[\r]", "\\1-\\2-\\3", "g", $2)
    # print date
}

# this represents a timed entry with date and time stamp YYYYMMDDTHHMMSS
# we ignore the seconds

/^DTSTART:/ {
    # print $0
    date = strftime("%Y-%m-%d %H:%M", datetimestamp($2));
    # print date;
}

# and the same for the end date; here we extract only the time and append this to the 
# date+time found by the DTSTART entry.  We assume that entry was there, of course.
# should probably add some error checking here!  In time...

/^DTEND:/ {
    # print $0
    time2 = strftime("%H:%M", datetimestamp($2));
    date = date "-" time2;
}

# The description will the contents of the entry in org-mode.
# this line may be continued.

/^DESCRIPTION/ { 
    $1 = "";
    entry = entry "\n" gensub("\r", "", "g", $0);
    indescription = 1;
}

# the summary will be the org heading

/^SUMMARY/ { 
    $1 = "";
    summary = gensub("\r", "", "g", $0);
    insummary = 1;
}

# the unique ID will be stored as a property of the entry

/^UID/ { 
    $1 = "";
    id = gensub("\r", "", "g", $0);
}

# when we reach the end of the event line, we output everything we
# have collected so far, creating a top level org headline with the
# date/time stamp, unique ID property and the contents, if any

/^END:VEVENT/ {
    # translate \n sequences to actual newlines and unprotect commas (,)
    print "* " gensub("\\\\,", ",", "g", gensub("\\\\n", " ", "g", summary))
    print "  :PROPERTIES:"
    print "  :ID:       " id
    print "  :END:"
    print "  <" date ">"
    # for the entry, convert all embedded "\n" strings to actual newlines
    print ""
    # translate \n sequences to actual newlines and unprotect commas (,)
    print gensub("\\\\,", ",", "g", gensub("\\\\n", "\n", "g", entry));
    print "** COMMENT original iCal entry"
    print gensub("\r", "", "g", icalentry)
    summary = ""
    date = ""
    entry = ""
    icalentry = ""
    indescription = 0
}

# Local Variables:
# time-stamp-line-limit: 1000
# time-stamp-format: "%04y.%02m.%02d %02H:%02M:%02S"
# time-stamp-active: t
# time-stamp-start: "Last change:[ \t]+"
# time-stamp-end: "$"
# End:

[-- Attachment #3: Type: text/plain, Size: 75 bytes --]

-- 
Eric S Fraga
GnuPG: 8F5C 279D 3907 E14A 5C29  570D C891 93D8 FFFC F67D

[-- Attachment #4: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [google] updated awk script for google to org conversion

[Daniel Martins]

Eric,

It is working (up to the moment) perfectly!!

I am really grateful to you!

Daniel

2010/7/9 Eric S Fraga <ucecesf@ucl.ac.uk>:
> I've updated the awk script for converting Google calendar iCal export to org:
>
> - the time zone calculation is now down completely within the awk
>  script, not requiring any external calculations.
>
> - fixed handling of continuation lines as Google seems to have
>  changed, since last night, the format of the export iCal entries.
>
> Again, comments, questions and criticisms all more than welcome!
>
> Please note that the shell command required to use this is now much simpler:
>
>  awk -f thisawkscript < icsfile > orgfile
>
> as there is no longer any need to define the SECONDS variable.
>
> Enjoy! :)
>
> eric
>
> --
> Eric S Fraga
> GnuPG: 8F5C 279D 3907 E14A 5C29  570D C891 93D8 FFFC F67D
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>
>

Re: [google] updated awk script for google to org conversion

[Sven Bretfeld]

Hi Eric

Thank you very much for the script and the idea. I'm a fan of MobileOrg
for Android, but the diary functions are not convenient to deal with at
the moment. So I adopted your method for date handling. Some
observations:

1. wget doesn't overwrite an older basic.ics laying in the same folder.
   Instead it names the files of further downloads to basic.ics.1,
   basic.ics.2 and so on. Therefore I added an additional line (rm $ICS)
   to your shell-script (not the awk-script, but the one with the
   automatized procedure you sent in a previous posting):

--8<---------------cut here---------------start------------->8---
#!/bin/bash

ICS=/home/sven/bin/google2org/basic.ics
ORG=/home/sven/aktuell/myconf/googlecalendar.org
AWK=/home/sven/bin/google2org/ical2org.awk

cd /home/sven/bin/google2org/
wget http://www.google.com/calendar/ical/[...]/basic.ics

timezone=$(date +%z | sed 's/^\([+-]\)0/\1/')

seconds=$(($timezone*36))

awk -f $AWK < $ICS > $ORG

rm $ICS
--8<---------------cut here---------------end--------------->8---

2. Running the above script with cron, makes Emacs continually ask if
   googlecalendar.org should really be edited, because the file had
   changed on disk. So I put the whole thing into an Emacs function:

--8<---------------cut here---------------start------------->8---
(defun my-org-to-google ()
  (interactive)
  (save-excursion
  (find-file "~/aktuell/myconf/mygtd.org")
  (org-export-icalendar-this-file))
)

(defun my-google-to-org ()
  (interactive)
  (save-excursion
    (kill-buffer "googlecalendar.org")
    (shell-command
     (format
      "sh /home/sven/bin/google2org/importgoogle.sh")))
)

(if (string-equal system-name "kamaloka")
    (run-at-time "3:00am" nil 'my-org-to-google)
    (run-at-time nil 3600 'my-google-to-org)
)
--8<---------------cut here---------------end--------------->8---

Every night a new ical-export is created from my main org-file at
3:00am, waiting to be uploaded in the morning (do you know if it's
possible to automatically import this to GoogleCL?). Every hour the new
entries of GoogleCL are imported to googlecalendar.org (which I include
my agenda-files, of course). To avoid problems, this happens only on one
of my PCs (kamaloka).

Greetings,

Sven

Re: [google] updated awk script for google to org conversion

[Rémi Vanicat]

"Sven Bretfeld" <sven.bretfeld@gmx.ch> writes:


[...]

>
> 2. Running the above script with cron, makes Emacs continually ask if
>    googlecalendar.org should really be edited, because the file had
>    changed on disk. So I put the whole thing into an Emacs function:
>

you could use auto-revert-mode:

  Toggle reverting buffer when file on disk changes.

  With arg, turn Auto Revert mode on if and only if arg is positive.
  This is a minor mode that affects only the current buffer.
  Use `global-auto-revert-mode' to automatically revert all buffers.
  Use `auto-revert-tail-mode' if you know that the file will only grow
  without being changed in the part that is already in the buffer.


[...]


-- 
Rémi Vanicat

Re: [google] updated awk script for google to org conversion

[Michael Steiner]

On 2010-07-12 05:22, Rémi Vanicat wrote:
> "Sven Bretfeld"<sven.bretfeld@gmx.ch>  writes:
>
>
> [...]
>
>>
>> 2. Running the above script with cron, makes Emacs continually ask if
>>     googlecalendar.org should really be edited, because the file had
>>     changed on disk. So I put the whole thing into an Emacs function:
>>
>
> you could use auto-revert-mode:

That's what i do: i slightly changed ical2org.awk by replacing line 77

  print "#+TITLE:     Main Google calendar entries"

with

  print "#+TITLE:     Main Google calendar entries -*- mode: 
auto-revert; mode: org; -*-"

which does make it automatic exactly for this files.  Works fairly well. 
In addition as i only do one-way sync i also turn the files (via 
file-permissions) read-only so the buffer is read-only and not 
accidentally modifled ..

-michael-

BTW: Sven, if you use option -O <outputfilename> you don't have the 
problem .[0-9] files; besides it's easier to manage multiple calendars ...

Re: Re: [google] updated awk script for google to org conversion

[Sven Bretfeld]

Michael Steiner <michisteiner@verizon.net> writes:

> On 2010-07-12 05:22, Rémi Vanicat wrote:
>> "Sven Bretfeld"<sven.bretfeld@gmx.ch>  writes:
>> you could use auto-revert-mode:
>
> That's what i do: i slightly changed ical2org.awk by replacing line 77
>
>  print "#+TITLE:     Main Google calendar entries"
>
> with
>
>  print "#+TITLE:     Main Google calendar entries -*- mode:
> auto-revert; mode: org; -*-"

Thank you very much. I will try that.

Greetings, 

Sven

Re: [google] updated awk script for google to org conversion

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 1262 bytes --]

On 12 Jul 2010 09:49:29 +0200, "Sven Bretfeld" <sven.bretfeld@gmx.ch> wrote:
> 
> Hi Eric
> 
> Thank you very much for the script and the idea. I'm a fan of MobileOrg
> for Android, but the diary functions are not convenient to deal with at
> the moment. So I adopted your method for date handling. Some
> observations:
> 
> 1. wget doesn't overwrite an older basic.ics laying in the same folder.
>    Instead it names the files of further downloads to basic.ics.1,
>    basic.ics.2 and so on. Therefore I added an additional line (rm $ICS)
>    to your shell-script (not the awk-script, but the one with the
>    automatized procedure you sent in a previous posting):

Yes, you are completely correct.  I trimmed down my own shell script a
bit too much.  I do have a "rm ${ICS}" command just before the
wget...  Sorry about that.

> 2. Running the above script with cron, makes Emacs continually ask if
>    googlecalendar.org should really be edited, because the file had
>    changed on disk. So I put the whole thing into an Emacs function:

I think somebody else has already suggested auto-revert-mode, which is
what I use so I don't see this happening at all.

Anyway, hope it continues to work for you and please do give any other
feedback you may have.


[-- Attachment #2: Type: text/plain, Size: 75 bytes --]

-- 
Eric S Fraga
GnuPG: 8F5C 279D 3907 E14A 5C29  570D C891 93D8 FFFC F67D

[-- Attachment #3: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [google] updated awk script for google to org conversion

[Marcelo de Moraes Serpa]

Very cool. Someone should setup a tutorial on Google<->org sync. As of
now, the information is spread around and outdated.

Thanks!

Marcelo.

On Mon, Jul 12, 2010 at 12:50 PM, Eric S Fraga <ucecesf@ucl.ac.uk> wrote:
> On 12 Jul 2010 09:49:29 +0200, "Sven Bretfeld" <sven.bretfeld@gmx.ch> wrote:
>>
>> Hi Eric
>>
>> Thank you very much for the script and the idea. I'm a fan of MobileOrg
>> for Android, but the diary functions are not convenient to deal with at
>> the moment. So I adopted your method for date handling. Some
>> observations:
>>
>> 1. wget doesn't overwrite an older basic.ics laying in the same folder.
>>    Instead it names the files of further downloads to basic.ics.1,
>>    basic.ics.2 and so on. Therefore I added an additional line (rm $ICS)
>>    to your shell-script (not the awk-script, but the one with the
>>    automatized procedure you sent in a previous posting):
>
> Yes, you are completely correct.  I trimmed down my own shell script a
> bit too much.  I do have a "rm ${ICS}" command just before the
> wget...  Sorry about that.
>
>> 2. Running the above script with cron, makes Emacs continually ask if
>>    googlecalendar.org should really be edited, because the file had
>>    changed on disk. So I put the whole thing into an Emacs function:
>
> I think somebody else has already suggested auto-revert-mode, which is
> what I use so I don't see this happening at all.
>
> Anyway, hope it continues to work for you and please do give any other
> feedback you may have.
>
>
> --
> Eric S Fraga
> GnuPG: 8F5C 279D 3907 E14A 5C29  570D C891 93D8 FFFC F67D
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>
>