From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Schulte Subject: Re: [test] Mark tests with missing dependencies as "expected to fail" Date: Tue, 18 Oct 2011 10:22:34 -0600 Message-ID: <87fwiq2rsl.fsf@gmail.com> References: <87fwiqnclu.wl%dmaus@ictsoc.de> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([140.186.70.92]:45488) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RGCRA-0000eE-DM for emacs-orgmode@gnu.org; Tue, 18 Oct 2011 12:22:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RGCR3-0003vQ-4z for emacs-orgmode@gnu.org; Tue, 18 Oct 2011 12:22:44 -0400 Received: from mail-qy0-f176.google.com ([209.85.216.176]:43945) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RGCR3-0003vM-2n for emacs-orgmode@gnu.org; Tue, 18 Oct 2011 12:22:37 -0400 Received: by qyk30 with SMTP id 30so793386qyk.0 for ; Tue, 18 Oct 2011 09:22:36 -0700 (PDT) In-Reply-To: <87fwiqnclu.wl%dmaus@ictsoc.de> (David Maus's message of "Tue, 18 Oct 2011 06:32:45 +0200") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: David Maus Cc: emacs-orgmode@gnu.org David Maus writes: > Hi all, > > Currently tests with missing dependency are silently skipped -- it > might be worth changing this behavior to not skip them, but mark them > as expected to fail. You can do this in ERT by placing the > keyword :expected-result followed by either :passed or :failed before > the test's body. > > Benefit of this is that the tests w/ missing dependencies will show up > in the ERT result page (with a small letter f) but (obviously) don't > count as failures. > > The following macro is a first shot at a convenient way to define > tests with dependencies: > > #+begin_src emacs-lisp > (defmacro org-test-deftest-with-dependency (dependency &rest body) > (let* ((docstring (if (stringp (third body)) (list (third body)))) > (deftest (nthcdr (if docstring 3 2) body)) > (dependency-p (eval dependency))) > `(ert-deftest ,@(append (list (first body) (second body)) docstring) > ,@(if dependency-p > '(:expected-result :passed) > '(:expected-result :failed (error "Missing dependency"))) > ,@deftest))) > #+end_src > > Here DEPENDENCY is a simple form that evaluates to non-nil if the > dependency is met. If marking the tests this way is the agreed way to > go we can extend the syntax of a dependency to an alist with a > human-readable description of the dependency as key and a list of > forms that all must eval to non-nil as value. E.g. > > #+begin_src emacs-lisp > (defvar test-ob-R-dependencies > '(("R executable" (org-test-for-executable "R")) > ("Package ESS" (featurep 'ess)))) > #+end_src > > And change the expander code to map through this alist and as soon one > dependency evals to nil sets a variable `dependency-missing' to the > respective dependency. > > Any comments on this? > Hi David, I agree it would be preferable to note that not all tests are run when dependencies are missing, although I don't think it is extremely important. I think some version of the above would be worthwhile if it could be done in a file-wide manner (as are the current dependency checks) and wouldn't require duplicating the dependency check or changing every test form individually. Perhaps a file-local-variable could be used to expect failures for every form defined in the file? Cheers -- Eric -- Eric Schulte http://cs.unm.edu/~eschulte/