Earlier I blogged about Epresent, which is basically a piece of code for making Org-Mode suitable for preparing presentation slides. There are times when I can’t resist mentioning the innovative Racket programming language in a presentation. In those situations I tend to want to have syntax-highlighted Scheme code on my slides, and also to evaluate the code snippets and insert the results next to the code listing. This is apparently the sort of thing one can do with Org-Mode Babel, for a variety of languages.

When configuring Emacs to set things up I wasn’t familiar with Babel or any of the solutions for evaluating Scheme code under Emacs for that matter. After some looking at Babel and Inferior Lisp, I didn’t manage to configure Babel to invoke Racket for evaluating a code listing. Instead I resorted to replacing the Babel code for Scheme support (in the ob-scheme.el) with basically just the following code:

(defun org-babel-execute:scheme (body params)
  (let* ((tangle (cdr (assoc :tangle params)))
         (script-file 
          (if (string-equal tangle "no")
              (org-babel-temp-file "org-babel-" ".rkt")
            tangle)))
    (with-temp-file script-file
      (insert body))
    (let* ((pn (org-babel-process-file-name script-file))
           (cmd (format "racket -u %s" pn)))
      (message cmd)
      (shell-command-to-string cmd)
      )))

This solution creates a new Racket instance for every evaluation, and hence is not as efficient as an Inferior Lisp based solution (or similar), but it works, is more straightforward, avoids Racket issues such as specifying the correct module context for evaluating the code, and the evaluation context is always “clean” as a new Racket instance is used.

For something more powerful and interactive one might look at integrating Geiser with Babel, for instance, but then some more hackery of Epresent would be in order to support such interaction. The current design doesn’t allow for “slides” to be edited during presentations, not with the same (presentation friendly) text styling.

Text rendering in Emacs has been looking mighty good since 23.1, and this opens up possibilities to do even more in Emacs. For example, I recently came across something called epresent.el on GitHub, by Eric Schulte et al. The epresent.el Emacs Lisp file leverages Org-Mode to implement a simple presentation mode for Emacs. Using Org-Mode is, at least for me, faster than struggling with something like OpenOffice.org Impress.

Having discovered epresent.el, I proceeded to customize it to my tastes, and even tried it out by EPresenting my slides during a two-hour lecture that I recently gave at University of Helsinki. I’d say the solution works fairly well for less serious presentations, making for something more informal and fun. Having access to the power of Emacs and Org-Mode also makes it convenient to do more than just show slides, as one can for example embed shell commands that help demonstrate what is stated on a given slide.

I’ve made my customized version of epresent.el available on GitHub. For Eric’s version, along with a sample document, see eschulte / epresent, also on GitHub.

To run the code you will want to be using a fairly recent version of Org-Mode (I’m using 7.4), and Emacs version 23.

While it’s possible to load and use epresent.el directly, without setting it up in any way, I would suggest adding something to your Emacs configuration file to make EPresenting more convenient. In my case I first add the directory containing epresent.el to the Emacs load-path, and then execute

(autoload 'epresent-run "epresent")

(add-hook 'org-mode-hook
        (function
         (lambda ()
           (setq truncate-lines nil)
           (setq word-wrap t)
           (define-key org-mode-map [f3]
             'epresent-run)
           )))

This sets up some parameters to my liking, and adds a key binding for conveniently starting a presentation. To exit a presentation (and go back to normal Org-Mode) one can simply press “q”.

C++ is a good language in that it is widely supported and has a large ecosystem around it. As a result, it has a large number of libraries, including extensive cross-platform frameworks (such as Qt). There are also entire operating systems written in C++, of which Symbian is an example. All of this provides ample motivation to use C++ (or at least its libraries) from time to time.

However, as a language C++ can feel verbose and constraining at times. You often find that you’re repeating yourself, and can’t find a way around it by staying within the language unless you are a wizard in the use of CPP preprocessor macros and template metaprogramming. So you might for example end up creating custom (and fragile) preprocessors (such as Qt’s moc), or hacking together ad-hoc code generating Perl scripts.

One option to avoid repeating yourself is not to consider C++ purely as a source language (something to be written by hand), and rather manage your code using an advanced IDE that can crank out and refactor code based on your instructions. (You don’t edit Word documents with Emacs either, after all.) Another, less interactive and more reusable (by rerunning or copy/pasting and editing) approach would be to use a language that is more flexible than C++ itself and can be translated to readable C++ (in the vain of CoffeeScript to JavaScript translation, for instance). The end result presumably could be roughly the same regardless of whether produced using an IDE or a translator.

There are many languages that compile down to JavaScript. I’d like to see some more languages that compile down to C++ and are fairly interoperable with C++ APIs. Or technologies enabling the implementation of such languages, or the extension of C++ itself. Currently I know of hardly any such languages or technologies, certainly nothing of the kind of maturity already manifested by CoffeeScript. C, being the high-level assembly language of CPUs, is a popular target language, but C++ however is not.

For the time being one can at least imagine what kind of features one would like to add to C++, if it were possible with reasonable effort. A couple of weeks ago I started recording ideas for language constructs that would be translatable to readable C++.

Some days ago I released a little code generation utility that I have been using for well over a year in cranking out repetitive C++ code. Koog is—for lack of a better established term—a mixed-code generator. It is similar to other tools of its kind (such as Cog), but the only one that I know of that uses the Scheme language for specifying what code to generate.

The concept of mixed-code generation is simple. You generally maintain a source code file by editing it manually, but you wish to generate some part or parts of the code. You mark those code sections with markers recognized by the code generator, and you also include (potentially foreign) code that produces the code to go between those markers. To avoid confusing the host language compiler, you make sure that the markers and the code generation directives have syntax such that they are treated as comments in the host language. These comments are generally retained, so that the code can be regenerated when the code generation directive or data affecting it changes.

A simple if somewhat pointless example of this is the following snippet of C/C++/Java/etc. code:

int x = /***koog (display (+ 1 2)) ***//***end***/;

When Koog is invoked on the above input, it expands to:

int x = /***koog (display (+ 1 2)) ***/3/***end***/;

I largely use the tool interactively under Emacs, with custom libraries for generating code for some of the Symbian C++ idioms.

See the Koog homepage to download the tool, or to find out more about it.

I wrote a technical report about the ContextLogger2 software that I’ve been working on at HIIT of late. The direct URL and other publication details can be found from this page. The entire series of electronically published HIIT technical reports is available from here.

I’ve tried GCCE version 4 before for building Symbian software, but have been somewhat put off by all the warnings it produces, mostly due to incompatibilities with the header files in the existing S60 SDKs. Well, I finally managed to get a working S60 application built with GCCE 4 without any warnings. Mind you, this was just one application, and not built with the full Symbian toolchain, but still.

I used the S60 5th Edition SDK with a number of patches, some of which I have posted here. The version of GCCE that I used is Sourcery G++ Lite 2010q1-190, which is the latest at the time of writing, and corresponds to GCC version 4.4.1. (When downloading it, I’d recommend opting for a tarball of the binaries where suitable. These in my opinion are the easiest to install, as mere unpacking is required.)

There are blog posts and discussion regarding some of the compiler warnings that appear with GCCE 4. Patching an SDK to remove the warnings is considerably easier than the task the Software Freedom Fighters are faced with, as one need not be able to build Symbian OS, but just compile against it, and as there is no moving target, but rather whichever SDK release you want to use.

The Forum Nokia How to use GCCE 4 with Symbian SDKs wiki page lists a number of reasons as to why a switch to version 4 of GCC might be beneficial. However, for those like me who have previously been sticking with the 2005q1 release (GCC version 3.4.3) of GCCE that the S60 SDKs have been shipping with as standard (and still do), perhaps the most important benefit of switching to a more recent compiler is that it should finally be possible to have static writable data in DLLs. The bug preventing that has apparently been fixed by now.

An additional benefit is that a lot of the features in the upcoming C++0x standard should be available by invoking the compiler with the -std=c++0x switch. Refer to this table to see which features are supported by GCC 4.4.

Symbian’s ABLD build system tends to produce a lot of output, especially when run with the verbose flag (-v), as I like to do to see the compiler invocations. Also, ABLD likes to run to “completion” despite individual build steps failing. The end result is that it takes some effort to look for the errors and warnings in the output.

Some suggest scanning ABLD output in order to terminate the build on error or to highlight errors and warnings. I like to use a combination of these methods, by stopping the build on any error, and by highlighting any warnings in a different (red, in my case) color.

To implement such functionality, one can drive abld through a script

$ abld-filter.rb abld build -v gcce urel

where the abld-filter.rb script might be something like

#!/usr/bin/env ruby

def chcolor(num)
  "\033[#{num}m"
end

def red(text)
  "#{chcolor(31)}#{text}#{chcolor(0)}"
end

WARNING_RE = /warning/i
ERROR_RE = /(?:fatal error|^make.*Error|error: )/i

def sh_check_build_errors(cmd)
  IO::popen(cmd + ' 2>&1') do |input|
    input.each_line do |line|
      line.chomp!
      if line =~ WARNING_RE
        puts(red(line))
      else
        puts(line)
      end
      if line =~ ERROR_RE
        puts "build error"
        exit 1
      end
    end
  end
end

unless ARGV.empty?
  command = ARGV.join(" ")
  sh_check_build_errors(command)
end

With the highlighting set up, one might also want to intentionally produce warnings, just because they look so colorful. To point out to-do items using the C/C++ preprocessor, for instance.

#warning TODO: error checking to be implemented

Earlier I wrote about a problem with restoring data to my Nokia E71 after a firmware upgrade. Well, there was also another problem. Once again, I started getting the “Untrusted software found on memory card. To install, go to Application manager.” complaint upon my Nokia E71 booting or otherwise mounting the memory card. Application Manager showed no applications marked as “Not installed”, and reinstalling the applications I tend to have installed didn’t fix the problem.

I had reason to suspect that the problem was one or more untrusted, previously installed themes, but the question there was how to find and remove them. S60 provides no function (that I know of) for removing applications and themes that for whatever reason do not show up in Application Manager, and platform security prevents direct access the relevant parts of the filesystem. One has the option of either cracking platfrom security or mounting the memory card as mass storage on another system. I opted for the latter.

The e:/private/10207114/import directory appears to have all the installed themes, while the e:/private/10202dce directory will have any uninstalled ones. The names of the SIS files there are not particularly descriptive, but a tool such as sisinfo can be used to look at the contents of the files, making it easy to determine which ones contain a theme. I used the command

for file in *.sis; do echo $file; sisinfo -f $file -i; done

to list the contents of all the SIS files, and then proceeded to delete all the themes. This got rid of the complaint about untrusted software.

My E71 is now fully recovered from the v300 firmware upgrade, just in time for the release of v400 firmware.

The other day I gave a presentation on the history of the programming language Lua, in a HOPL conference inspired seminar course at TKK. If you asked a presentation expert, you’d probably be told that the slides have too many words in them, but perhaps that will make them easier to follow without the accompanying talk. In any case, the presentation slides have been posted on the web.

So I upgrade the firmware of my Nokia E71 from v200 something to v300 something.

After the upgrade, I proceed to restore my data, as one would. No errors reported, and indeed everything was restored, with the minor exception of all of my 1540 Calendar entries, which were nowhere to be seen. The same thing with restore from memory card and restore with Nokia PC Suite. Sometimes I wish something this important would just work.

Naturally, I then had little choice but to proceed to extract the vCalendar 1.0 data from the .nbu file directly, using a little PLT Scheme library that I hastily hacked together for this purpose (copy/pasted below for your convenience). I then sent the resulting file to the phone, and it indeed took some time to transfer, but then, nothing. Nothing in the Inbox, no notification of any kind.

I then tried splitting the file to five smaller chunks, and sending these individually. These smaller files did appear in Inbox, and would actually import to the calendar database. And I once again have fresh firmware and am able to see my appointments for next week.

Nokia’s customers have so many different combinations of devices, firmware versions, PC Suite versions, and so much data, that no way is everything going to always work flawlessly. Would it be such a bad idea to have descriptive error messages to help resolve any problems? Yes, I know error handling and reporting is hard work, but probably something that can be solved with sufficient manpower. Writing error-free software is considerably is harder still.

#lang scheme

#|

Example REPL use:

> (require "nbu-extract-vcal.ss")
> (vcal-extract "2009-11-07 Nokia E71.nbu")

|#

(provide vcal-extract)

;; Returns the number of entries extracted.
(define (vcal-extract nbu-fn)
  (define count 0)

  (define entry-re #px#"BEGIN:VCALENDAR.*?(BEGIN:.*?)END:VCALENDAR")

  (define (f in out)
    (let loop ()
      (let ((res (regexp-match entry-re in 0 #f)))
        (when res
          (let ((entry (second res)))
            (write-bytes entry out)
            (write-bytes #"\r\n" out)
            )
          (set! count (+ count 1))
          (loop)))))

  (let ((vcal-fn (string-append nbu-fn ".vcal")))
    (call-with-input-file
      nbu-fn
      (lambda (in)
        (call-with-output-file
          vcal-fn
          (lambda (out)
            (write-bytes #"BEGIN:VCALENDAR\r\nVERSION:1.0\r\n\r\n" out)
            (f in out)
            (write-bytes #"END:VCALENDAR\r\n" out)
            )
          #:exists 'truncate/replace))))

  count)

Not quite up there with “Keyboard not found. Press F1 to continue.”, but…

So I wanted to use my Nokia E71 for something, but first had to try dismissing this dialog, but of course first I would have needed to remove the keylock, but hey, it was the keylock controlling application that had crashed.

S60 3rd Edition has been out for quite a while by now, and for me, the one thing that has all but killed the pleasure of developing native software for the platform has been the humiliating wait—dismiss dialog—wait—dismiss dialog—wait—dismiss dialog—wait procedure during installation of the software being tested and developed. For one-off installations the installation procedure is no problem, but when you’re a developer it gets old real fast.

On 1st and 2nd edition I would bypass the installer by simply transferring the binaries to where they ultimately belong on the target filesystem, using a tool such as obexftp. The platform security introduced in 3rd edition has prevented access to the relevant directories, making the same approach impossible. (Even on a hacked phone, where transferring files to protected directories is possible, the system still checks the checksums of the binaries, refusing to run software that did not go through the installer, it seems.)

While I’m all in favor of protecting the end-user from malware with a platform-wide security architecture (rather than security being something of an afterthought), I’m firmly of the opinion that a legitimate developer should be able to disable any and all security features that hinder development, on the devices that he/she uses for development. Whether this means buying the devices through special channels, having special firmware flashed at a service center, or installing manufacturer-signed IMEI-bound PlatSec disabler software, or whatever.

The inconvenient installation of native code has been a clear disadvantage for native S60 development when compared to developing with Python for S60, say, as the Python interpreter does not forbid the loading of Python code from “insecure” directories.

Nokia surely has some functionality in their collection of development tools that can automate the installation of native software, but I’m not aware of the S60 SDKs or the Carbide.c++ IDE including any such solution that could be integrated in other environments and toolchains, at least not in a way that is documented. Some months ago I finally decided to address this issue, and ended up creating such a solution. It may well be the most useful piece of Symbian software I have ever written. Finally, finally a tolerable developer experience when doing native S60 development.

The software, a Python for S60 extension named pyswinst, is available via a dedicated webpage hosted at HIIT. The included auto_installer.py example is a complete program that can be used to automate SIS file installations, when used together with any suitable file transfer tool. The pyswinst extension might conceivably even handle JAR files or WRT widgets, which are also supported by the Application Manager application, but I haven’t tried.

Installation using this software is not exactly fast, but at least it’s convenient, and lets you do other things while waiting for an installation to complete. Let your makefile take care of transferring and installing the software you develop. Start today.

Earlier releases of the Miso PyS60 extension had a take_photo function, for taking photos (without a viewfinder). This functionality (and more) has since then been integrated to PyS60 itself in the form of the built-in camera module.

I’d now like to introduce the pynewfile library for PyS60, which (at least for some S60 phones) offers an alternative to the camera module when wishing to take photos interactively with a viewfinder.

Now, I’m not quite sure what exactly it is that the pynewfile module does, but it can apparently launch the S60 Camera application in some kind of “server mode” (or slave mode), having it run just long enough for the user to take and select a photo.

The difference between using camera and pynewfile then is that the latter allows the user to use the hopefully familiar Camera application view and its controls when taking a photo, whereas with camera it is the programmer who has to place the viewfinder and implement any necessary controls. Which alternative is more desirable depends on the application.

pynewfile is available via a dedicated webpage hosted at HIIT.

Early last year at HIIT I hacked the Python for S60 inbox module to have it support querying for the content and metadata of messages in the device Inbox that have been sent via Bluetooth OBEX Push. (The built-in inbox module apparently only supports accessing SMS messages.) I changed the name of the hacked module to pyinbox, to avoid conflicts with the built-in Inbox access module.

I recently made pyinbox available via a dedicated webpage, in case it should prove useful to others.

In the PDIS project at HIIT we developed a socket library for Python for S60. The library consisted of both a native extension and quite a bit of Python code. The native extension was called aosocketnativenew, and became somewhat well known as an extension supporting non-interactive Bluetooth discovery.

We never made a standalone release of the library, nor did we release a version of the native extension built for S60 3rd Edition. Until now, anyway. I recently happened to have use for the extension again, and ended up building it for S60 3rd Edition, adding support for access point selection for outbound TCP connections, and making a release of just the extension. The extension is now called pyaosocket rather than aosocketnativenew. Source and binaries are available via a dedicated webpage hosted at HIIT, but there still is no API documentation.

I am a big fan of Python’s keyword argument facility, and especially its support for *args and **kwargs function parameter declarations. *args and **kwargs capture any explicitly undeclared positional and keyword arguments, respectively, and this facility in many cases allows one to avoid repeating function interfaces. This is both less typing and more future proof, and makes it easier to see the parameters that directly concern a function. Contrast this with Java, which not only has no keyword arguments, and no support for *args style declarations, but which also forces you to repeat caught exception declarations in function signatures.

In PLT Scheme version 3, there was no support for keyword arguments built into the language as such, but there was a handy kw.ss module for adding support for keyword arguments. Thanks to the language extensibility enabled by macros, the kw.ss-provided keyword argument support was (and is) a pleasure to use, and I’ve accumulated a lot of code making use of kw.ss.

When using kw.ss, keyword arguments are passed like any other arguments, but it is possible to declare “special” functions that actually interpret the passed keyword arguments. A common idiom I would use is shown by the following code:

(define (create-file filename data . rest)
  (verbose-puts (format "write file ~a" filename))
  (unless (simulate)
    (apply
     call-with-output-file
     filename
     (lambda (output)
       (display data output))
     rest)))

Here, the create-file function only cares about the first two positional arguments filename and data, and passes anything else—whether positional or keyword arguments—to call-with-output-file. So the rest bit serves both as *args and **kwargs, and we need not know whether call-with-output-file takes its options as positional or keyword arguments, or both. In fact in PLT Scheme v3 it takes them as positional arguments, while in the recently released PLT Scheme version 4 it takes them as keyword arguments. One might hope that the above code (which I wrote for v3) would be compatible both with v3 and v4, but unfortunately this is not quite the case.

PLT Scheme v4 has a kw.ss-incompatible keyword argument implementation built into the language, and while PLT Scheme has admirable support for backward compatibility, for many modules it probably makes sense to switch to the v4 scheme language eventually, which then would involve tweaking any keyword argument passing. Somewhat surpringly (to me), even the above code is not compatible with the scheme keyword arguments, even though it does not explicitly mention such arguments.

If a function does not look like it accepts keyword arguments, and yet it is invoked with keyword arguments, the compiler will complain. For instance, in the following code the invocation of foo with a keyword argument is an error, while invoking it without arguments would be acceptable.

(define (bar #:a (a 1))
  (write-nl a))

(define (foo . op)
  (apply bar op))

(foo #:a 555)

Now, v4 presently has no easy way to declare **kwargs, and the question then is how to implement the pass-on-keyword-arguments idiom in v4 without too much typing. Luckily, it seems that macros can offer pain relief in almost any situation where the language given to you is in some way tedious to use. My initial solution to passing along keyword arguments, and a v4 version of create-file is given below. Naturally, if desired, it ought to also be possible to implement a define/pass-kw macro, and a kw.ss compatible version of lambda/pass-kw.

(define-syntax lambda/pass-kw
  (syntax-rules ()
    ((_ arg-spec apply/kw body ...)
     (make-keyword-procedure
      (lambda (kw-names kw-vals . args)
        (let ((apply/kw
               (lambda (f . rest)
                 (keyword-apply f kw-names kw-vals rest))))
          (apply
           (lambda arg-spec body ...)
           args)))))))

(define create-file
  (lambda/pass-kw (filename data) apply/kw
    (verbose-puts (format "write file ~a" filename))
    (unless (simulate)
      (apply/kw
       call-with-output-file
       filename
       (lambda (output)
         (display data output))))))

There are a few Symbian software authors who occasionally write a piece of Symbian code, possibly with no intention of ever touching the code again, package it up as a SIS file, and then “fire and forget” that SIS file to some web site in case someone should find it useful. Or forget until such time that someone complains about the SIS file having been expired.

Some of the blame here falls to Symbian’s makekeys tool, which by default generates certificates with the validity period of only one year. In fact, apparently only more recent versions of the tool support generating longer-lived certificates. It is possible, though, to use “standard” tools such as openssl to generate certificates. For instance, to generate a key and self-signed certificate valid for 10 years, one can use a command such as

openssl genrsa 1024 > my.key
openssl req -new -x509 -nodes -sha1 -days 3650 -key my.key > my.cer

Anyway, given how common the “expired software” is in the Symbian world, it would often be useful to find out when exactly it is that a SIS file expires. Either to find out by how much to turn back time on your device to allow a SIS file with an expired signature to install, or to determine if your own SIS files have expired or are about to expire soon. However, I’ve yet to come across any tool that will actually tell you the expiration date, given a SIS file.

When one has a certificate generated as above, it seems to be possible to see the date (in a somewhat readable form) with the command

openssl asn1parse -inform PEM -in my.cer

You will want to look for two entries called UTCTIME, which specify the start and end time of the validity period.

SIS files signed with a certificate do contain the signing certificate in them, albeit in the different DER (binary) format. To get the certificate extracted and the dates printed out, one might say hack the very nice SISInfo tool a bit.

In SISInfo, there is a bit of code that reads “blob” fields (see the SISBlobField class), and one of those fields contains the certificate. An ugly hack to extract the dates is as follows:

def readableStr(self):
        fp = open("/tmp/siscert", "w+b")
        try:
                fp.write(self.data)
        finally:
                fp.close()

        pipe = os.popen("openssl asn1parse -inform DER -in /tmp/siscert", "r")
        res = ""
        try:
                for line in pipe:
                        ix = line.find("UTCTIME")
                        if ix != -1:
                                res += line[ix:len(line)]
        finally:
                pipe.close()
                fp.close()
        return res

The certificate used to sign v1.08 had expired, so created a longer lived one with openssl, and signed with that. Added the backup registration file while at it, as it had been missing from previous 3rd edition releases.

Some operations on the Nokia E61 are painfully slow. For instance navigating the applications menu can be time consuming, especially if the folders being opened have many items in them. The other day I managed—by (un)installing some SIS files—to get the icon server of the device to crash (it produced a panic). This resulted in an iconless applications menu, with only application names visible, but otherwise seemed to have no adverse effect. And navigating the menu had become much faster, with no folder taking more than a fraction of a second to open.

The app menu navigation on Nokia E71, the latest Nokia crackberry, seems much snappier than on the E61, and incidentally the E71 icons also look simpler than the ones we’re used to seeing on recent S60 devices.

The SDK API Plug-in package for S60 SDKs has quite a few interesting APIs, such as the Browser Launcher API and the New File Service Client API.

Some days ago I wrote a script for installing plugins from the package onto GnuPoc style SDKs. It may be downloaded from http://www.hiit.fi/files/fi/da/gnupoc/.

When designing socket code to be implemented in C, it may well make sense to first sketch out the design in Ruby. The Socket class provides thin wrappings for just about all of the relevant C standard library functions. There also is the fcntl module, and select in Kernel. The socket code can thus be just about the same in Ruby as it would be in C, but trying out different designs for say managing sessions in a server is much less tedious in Ruby.

@ssock = Socket.new(Socket::AF_INET,
                    Socket::SOCK_STREAM,
                    Socket::IPPROTO_TCP)
sockaddr = Socket.pack_sockaddr_in(@port, 'localhost')
@ssock.setsockopt(Socket::SOL_SOCKET, Socket::SO_REUSEADDR, true)
@ssock.bind(sockaddr)