Category Archives: Code

FOP and ZXing compatibility

From the I’ve been trying to get this to work — this tip may save you some time department:

Apache FOP is for doing graphics with XML and XSLT.

FOP version 0.95 came out in August 2008. 1.0 in July 2010. 1.1 in October 2012.

ZXing is for doing QR codes.

ZXing 0.1.2 was released in March 2012, and seems to no longer be in active development.

The important bit: As far as I can see, ZXing works fine with FOP 0.95 (which was current when it was released), but not with later versions.

I kept getting these errors:

SEVERE: Image not available. URI: (instream-object). Reason: org.apache.xmlgraphics.image.loader.ImageException: The file format is not supported. No ImagePreloader found for null (No context info available)
org.apache.xmlgraphics.image.loader.ImageException: The file format is not supported. No ImagePreloader found for null

With any amount of fiddling, I couldn’t get it to work on Windows and Java 1.7. Nobody online appears to have noted this issue. You might have more luck…

Or maybe there is no solution to it. If so, I hope this saves you some time.

(Barcode4J, which we also use, seems to work with all currently available versions of FOP. It also has QR functionality, provided via ZXing, but this is only available in the unreleased code, which presumably has alpha status, and you have to build yourself. Barcode4J also hasn’t been updated in some years, last released in December 2010.)

Unhelpful web help

Just… just… wrong. So wrong.

FlickrHelp
Firstly, note the error message “Enter a valid email addresss”. Where, pray tell, ought I do this?  Why do I need to upload any attachment again?  Why do I have to prove I’m a human time-after-time, when all I’m doing is wrestling with your completely broken attempt at a web form?

Have they noticed that no-one is submitting help requests via this form, what with its refusal to accept said requests?

Dear Flickr: stop sucking balls.

Political donations are not the problem

Corrupt politicians have recently been in the Australian news.

It has been observed that money, in the form of political donations, is a corrupting influence. This causes hand-wringing, as banning donations is considered to hinder the freedom of political expression.

As a response to this demand for cash to finance political expression, suggestions are made that private funding of politics be replaced by public funding – basically an increase on the funding which parties already receive (something of the order of $2.48 per primary vote in lower house seats in the last federal election, for example). This grates those with a strong dislike of politicians and the political process. In addition, the current funding model of retrospective funding (based on votes received) disenfranchises new political views – it locks in the existing players by funding them, allowing them to campaign for votes that will fund them; those outside the system will not be able to break in.

To allow new entrants into the political system to be funded on an equitable basis, some kind of on-going polling could be done and a funding stream allocated on proportionate support in non-electoral polls.

However, switching to purely taxpayer-funded funding isn’t necessary, even if in effect the tax-deductibility of political donations makes them taxpayer subsidised.

Political donations are not the problem, the problem is that donors can be identified by the political party and and expectation of quid pro quo is raised. Beyond that, large donations from a single donor are also a problem – even if political party donations were anonymised and repudiable the donation’s existence could be inferred by the velocity of money flowing out of any anonymising system.

Let’s say you’re trying to run a corrupt political party under an annoymised donation system. Someone comes to you and says “I will give your corrupt party $10m, and I expect you to make this corrupt thing happen.” You’d then donate the $10m, and your donation would be pooled along with the hundreds of other donations made to the party. The Donor Anonymising Service (DAS) would then hand over a certain amount of money to the party, but it would not be $10m. It would be the stipend that the party had requested from the DAS, along with advice that the current amount held in reserve is enough to last at least X days, where X was the same number (give or take a couple of days) as it was yesterday. You don’t know if the $10m donation was actually made, all your party knows is that it’s got enough money to last X+2 days. You could up the rate of the stipend, but the DAS would scale back the reported window so that no extra information is revealed by the reported minimum duration the reserves will last. You’d limit the rate and number of times the stipend could be changed to discourage probing. Naturally, it would be illegal to make a political party aware of a donation or its amount.

Of course, then you have all the fun and games associated with loaning money to political parties, and with corrupt administration of a Donor Anonymising Service, but you get the gist of where we could go with this idea.

BASIC turns 50

The BASIC programming language turned 50 on Thursday.

This Time article is a great read — and notes the importance of the language on getting school students programming.

What’s the equivalent today? The Raspberry Pi is helping make hardware affordable. Some might dabble in Visual Basic or C# via Visual Studio Express, or the many of the other freely available languages such as Python, PHP, Javascript… or here’s another way of looking at it, from Jeff Atwood:

Programmatically create Django security groups

Django authentication has security roles and CRUD permissions baked in from the get-go, but there’s a glaring omission: those roles, or Groups, are expected to be loaded by some competent administrator post-installation.  Groups are an excellent method of assigning access control to broad roles, but they don’t seem to be a first-class concept in Django.

It seems that you can kind-of save these values in by doing an export and creating a fixture, which will automatically re-load at install time, but that’s not terribly explicit – not compared to code. And I’m not even sure if it will work.  So here’s my solution to programmatically creating Django Groups.

management.py, which is created in the same directory as your models.py and is automatically run during python manage.py syncdb:

from django.db.models import signals
from django.contrib.auth.models import Group, Permission
import models 

myappname_group_permissions = {
  "Cinema Manager": [
    "add_session",
    "delete_session",
    "change_ticket",
    "delete_ticket",         # for sales reversals
    "add_creditcard_charge", # for sales reversals
    ],
  "Ticket Seller": [
    "add_ticket",
    "add_creditcard_charge",
    ],
  "Cleaner": [ # cleaners need to record their work
    "add_cleaning",
    "change_cleaning",
    "delete_cleaning",
    ],
}

def create_user_groups(app, created_models, verbosity, **kwargs):
  if verbosity>0:
    print "Initialising data post_syncdb"
  for group in volunteer_group_permissions:
    role, created = Group.objects.get_or_create(name=group)
    if verbosity>1 and created:
      print 'Creating group', group
    for perm in myappname_group_permissions[group]: 
      role.permissions.add(Permission.objects.get(codename=perm))
      if verbosity>1:
        print 'Permitting', group, 'to', perm
    role.save()

signals.post_syncdb.connect(
  create_user_groups, 
  sender=models, # only run once the models are created
  dispatch_uid='myappname.models.create_user_groups' # This only needs to universally unique; you could also mash the keyboard
  )

And that’s it. Naturally, if the appropriate action_model permissions don’t exist there’s going to be trouble.  The code says: After syncdb is run on the models, call create_user_groups.

Where did I take that photo?

I couldn’t find anyone extracting out the geolocation geotagging EXIF data from their photographs so they could pull it up on something like Google Maps.  There are stand-alone programs with embedded maps, but the bits and bobs lying around on the average system ought to be enough to just generate a URL to a mapping website.  The following bash script echoes the  URL that geolocates your JPEG.  Because my camera doesn’t emit it, I couldn’t be bothered dealing with the seconds part of a location, but I did detect that you don’t have a camera the same as mine.  Drop a line if you’ve used this and fixed it.

#!/bin/bash
# emit a hyperlink to google maps for the location of a photograph
declare Seconds=""
Seconds=`exif -m --ifd=GPS --tag=0x02 $1 | grep -oP "[\d|\d\.]+$"`
if (( $Seconds=='0' ))
then
  Seconds=`exif -m --ifd=GPS --tag=0x04 $1 | grep -oP "[\d|\d\.]+$"`
fi
if (( $Seconds!='0' ))
then
  echo
  echo "Script does not support seconds being specified"
  exit
fi
echo -n "https://maps.google.com.au/?q="
declare NorthSouth=`exif -m --ifd=GPS --tag=0x01 $1`
if [ "$NorthSouth" == "S" ] 
then
  echo -n "-"
fi
echo -n `exif -m --ifd=GPS --tag=0x02 $1 | grep -oP "^[\d|\d\.]+"`
echo -n "%20"
echo -n `exif -m --ifd=GPS --tag=0x02 $1 | grep -oP "(?<= )[\d|\d\.]+,"`
declare EastWest=`exif -m --ifd=GPS --tag=0x03 $1`
if [ "$EastWest" == "W" ]
then
  echo -n "-"
fi
echo -n `exif -m --ifd=GPS --tag=0x04 $1 | grep -oP "^[\d|\d\.]+"`
echo -n "%20"
echo -n `exif -m --ifd=GPS --tag=0x04 $1 | grep -oP "(?<= )[\d|\d\.]+(?=,)"`
echo

Install mwparserfromhell on Linux

Here’s how to install mwparserfromhell on Linux:

sudo apt-get install python-dev
sudo apt-get install python-pip
git clone https://github.com/earwig/mwparserfromhell.git
cd mwparserfromhell
python setup.py install

After which, wikitools by MrZ-man is nice for power-users:
svn co https://github.com/alexz-enwp/wikitools
cd wikitools/trunk
sudo python setup.py install

Installing Pygal into Cygwin

Pygal is a python library for emitting SVG charts. It might do PNGs too; the documentation is… sparse. Okay, there’s no documentation, but they show you several ways to make bar charts, and figure you can follow on from there.  Anyways, the installation instructions don’t work, not under cygwin.

Here’s what you should do:

  1. ensure cygwin has the libs libxml2-devel and libxslt-devel installed
  2. issue the command
    cygwin$ pip install pygal

and you’re done. Getting pip into cygwin is a whole world of hurt, but you will need to go looking for a http (not https) source to download setuptools, then download and run ez_setup.py, followed by using pip to upgrade setuptools. Have fun with that; I know I did.

Allow more JavaScript, maintain privacy

I’ve long regarded JavaScript in the browser to be one of the biggest security holes in web-browsing, and at the same time the Internet works less and less well without it. In 2008 Joel Spolsky made the observation that for some people the Internet is just broken:

Spolsky:   Does anybody really turn off JavaScript nowadays, and like successfully surf the Internets?

Atwood:   Yeah, I was going through my blog…

Spolsky:   It seems like half of all sites would be broken.

Which is not wrong.  Things have changed in the last five years, and now the Internet is even more broken if you’re not willing to do whatever random things the site you’re looking at tells you to, and whatever other random sites that site links off to tell you to, plus whatever their JavaScript in turn tells you to. This bugs me because it marginalizes the vulnerable (the visually impaired, specifically), and is also a gaping security hole.  And the performance drain!

Normally I rock with JavaScript disabling tools and part of my tin-foil-hat approach to the Internet, but I’m now seeing that the Internet is increasingly dependent on fat clients. I’ve seen blogging sites that come up empty, because they can’t lay out their content without client-side scripting and refuse to fall back gracefully.

So, I need finer granularity of control.  Part one is RequestPolicy for FireFox, similar to which (but not as fine-grained) is Cross-Domain Request Filter for Chrome.

The extensive tracking performed by Google, Facebook, Twitter et al gives me the willys. These particular organisations can be blocked by ShareMeNot, but the galling thing is that the ShareMeNot download page demands JavaScript to display a screenshot and a clickable graphical button – which could easily been implemented as an image with a href. What the hell is wrong with kids these days?

Anyway, here’s the base configuration for my browsers these days:

FireFox Chrome Reason
HTTPSEverywhere HTTPSEverywhere Avoid inadvertent privacy leakage
Self Destructing Cookies “Third party cookies and site data” is blocked via the browser’s Settings, manual approval of individual third party cookies. Avoid tracking; StackOverflow (for example) completely breaks without cookies
RequestPolicy Cross-Domain Request Filter for Chrome Browser security and performance, avoid tracking
NoScript NotScripts Browser security and performance, avoid tracking
AdBlock Edge Adblock Plus Ad blocking
DoNotTrackMe DoNotTrackMe Avoid tracking – use social media when you want, not all the time
Firegloves (no longer available), could replace with Blender or Blend In I’ve have had layout issues when using Firegloves and couldn’t turn it off site-by-site

ABC’s Labor bias: Vote Compass

I am a single issue voter. I’m not proud to admit being so shallow, but there it is.  If there was a party that wanted to cut greenhouse gas emissions by 50% by 2020 and also drown kittens, I’d be in like Flynn – and not because I dislike kittens either. Perhaps it’s because I take a root-cause view of the world.  Immigration problems?  Address climate change or it’s going to get much, much worse. Not spending enough on education? Not much point in edumacation if the climate collapses around us and we’re up to our ears in climate refugees. Wrong telecommunications plan? Choosing between having enough food and downloading porn faster than you can watch it doesn’t seem to be much of a choice to me.

So, naturally I thought that the ABC’s Vote Compass wouldn’t have much trouble pigeonholing me. Except, it tells me my views align more closely with the ALP.  Although at one stage in its questioning it allows you to weight the importance of issues (which I gave as 1-3 for most, 4 for a couple and 10 for environmental) this clearly… doesn’t carry any weight. The anemic 5% by 2020 cut embraced by the two major parties means neither will get my vote, regardless of the technique to “achieve” such a “challenging” target.

And yet Vote Compass thinks I’d make a good ALP voter.  I think not.  The ABC’s Labour bias at it again.

Replace a missing remote control with an Arduino and a laptop

I recently found myself without a remote for my WDTV Live media player, and limited resources to do anything about it – but I did have an Arduino, a breadboard and the local Jaycar had an IR LED.  Controlling IR devices is common practice with an Arduino. I would even be able to hack in functions that didn’t exist on the manufacturer’s remote – like creating a three minute skip by switching to 16x speed for 12 seconds.

The first port of call was to obtain Ken Shirriff’s Arduino IR remote control protocol library – as opposed to communications protocols, of which there are quite a number; did you know the first cut of WiFi included an infrared version? Without the remote, I wasn’t able to record and playback the IR signals sent to the WDTVLive, as you would with a learning remote. I had to find what to transmit from my custom remote. I little googling and I found the WD TV Live infrared remote control codes, which also helpfully reveals that the protocol is NEC.

I knocked up a quick proof of concept, installed it and watched it not work. Given I can’t see in infrared, I didn’t know if my circuit was working. I hooked a red LED up in parallel, and it didn’t light up; I thought I had cathode and anode swapped around, so flipped the red LED – and it didn’t light up. I pulled the IR LED, and then the red LED worked… I was shorting out the red LED. I couldn’t – with the bits I had lying around – confirm the device was transmitting anything. Rather than put the LEDs in series, I got a cheap camera-phone with video function, and it could see IR just fine. And it turns out the IR LED was transmitting something, but the WD TV Live media player wasn’t listening. Why?

The NEC infrared control protocol transmits 32 bits in one of two formats, one old (as in elderly) format encodes for 256 devices with 256 commands each, and the other encodes for ~64K devices with 256 commands each. The first 16 bits encode the device, and the second 16 bits encode the command. 16 bits for one of 256 commands, you ask? Well, one byte of the second 16 bits is the command, and the other is – for error checking purposes – the one’s complement of that. Further details of the pulse timing and protocol contents are available in various places, but they neglect to mention the extended addressing format. There are many IR control protocols. To use Ken’s IR library you need to know which protocol is used (which the google search revealed), and you can determine the protocol from the timing data found in the LIRC definition of a protocol, in this case the LIRC infrared control protocol for WDTV Live media player remote. The LIRC protocol defintion format is described by WinLIRC, so you can see what the timings are. In this case, the NEC protocol is revealed by the header, one and zero definitions, along with the fact that each code has 16 bits of ‘pre-data’ and 16 bits of data (a 32 bit package). Everything I could see was showing that the two, separately arrived at sets of command codes that were empirically sampled from the real world were compliant with the spec. One of the things the spec taught me was to transmit the NEC code twice, and to wait 70ms between re-transmissions.

I wasted time finding other codes for the remote, in other formats; I checked for byte ordering issues. Nothing worked.

The actual problem was the unsigned long for the command was previously an int; failing to notice this simple error led me to spend a long time trying to figure out why nothing was happening when I transmitted a command. One of the problems with the C language is the guarantees about data sizes aren’t worth much.  My entire life has been spent programming on architectures that have 32 bit data words; C compilers on these machines have all defined an int as 32 bits, but I’ve always been aware that the language spec says that an int is at least as wide as a short, which is at least as wide as a char with actual widths being up to the compiler implementation (although why you’d have different words for things of the same size is beyond me).  The AVR microcontroller in question has an 8 bit word; mathematical operands typically yield an 8 bit result (multiply is an exception) with compilers needing to implement more instructions to yield greater data widths. The defines express the codes as four byte values, which were then wrangled into a two byte int, and then again into unsigned four byte integer when passed to the IR library. Truncated bits in a protocol like this were the cause of inactivity.

Even with this fundamental problem solved, confusion was added by the fact that one of the memory cells in my Arduino is faulty. Once IR control code transmission was working, I noticed that sometimes it didn’t work. I decided to echo the command to the serial port, and the command being transmitted didn’t match that for the key pressed – the second byte was wrong. I added code to work around this memory corruption (not shown in the code below, because this is a pretty unusual). I’ve never come across this kind of problem before, recognising and then solving something like that is pretty old-school.

/*
Pin 3 is hard-wired into the IR library as the emitter
 */
#include <IRremote.h>
//#define DEBUG

IRsend irsend;

#define btn_enter  0x219E10EF
#define btn_right  0x219E906F
#define btn_left   0x219EE01F
#define btn_down   0x219E00FF
#define btn_up     0x219EA05F
#define btn_option 0x219E58A7
#define btn_back   0x219ED827
#define btn_stop   0x219E20DF
#define btn_rew    0x219EF807
#define btn_ff     0x219E7887
#define btn_play   0x219E50AF
#define btn_prev   0x219E40BF
#define btn_next   0x219E807F
#define btn_eject  0x219E08F7
#define btn_search 0x219EF00F
#define btn_home   0x219E609F
#define btn_power  0x219E48B7

// Pin 13 has an LED connected on most Arduino boards.
// give it a name:
const int onboard_led = 13;
const int retransmit=2;
unsigned long play_after=0;

void setup()
{
  pinMode(3, OUTPUT);     
  pinMode(onboard_led, OUTPUT);     
  Serial.begin(9600);
  Serial.println("WDTV Live serial controlled IR remote");
  Serial.println("~ Power    Eject ^ & Search   Rew - + FF");
  Serial.println("  w         Back q e Enter   Play  P");
  Serial.println("a s d (Arrows)     x Stop    Last < > Next");
  Serial.println("3 - FastForward three minutes");
}

void loop() {
  unsigned long cmd=0;
  if (Serial.available()) {
    switch (Serial.read()) {
      case 'E':
      case 'e':
      case ')':
      case '0':
      case 'O':
      case 'o': cmd=btn_enter; break;
      case 'q':
      case 'Q': cmd=btn_back; break;
      case 'P':
      case 'p':
      case ' ': cmd=btn_play; break;
      case 'S':
      case 's': cmd=btn_down; break;
      case 'W':
      case 'w': cmd=btn_up; break;
      case 'A':
      case 'a': cmd=btn_left; break;
      case 'D':
      case 'd': cmd=btn_right; break;
      case '-':
      case '_': cmd=btn_rew; break;
      case '=':
      case '+': cmd=btn_ff; break;
      case ',':
      case '< ': cmd=btn_prev; break;
      case '.':       
      case '>': cmd=btn_next; break;
      case '/':
      case '?': cmd=btn_option; break;
      case '~': cmd=btn_power; break;
      case '!':
      case '1': cmd=btn_home; break;
      case '^':
      case '6': cmd=btn_eject; break;
      case '*':
      case '8': cmd=btn_search; break;
      case 'x':
      case 'X': cmd=btn_stop; break;
      case '3': 
        if (!play_after) play_after=4; break;
    }
  }
  if (play_after > 0) {
    if (cmd) {
      play_after=0;
    }
    else if (play_after > 5) {
      if (play_after < millis()) {
        cmd=btn_play;
        play_after=0;
      }
    }
    else {
      cmd=btn_ff;
      if (--play_after == 0) {
        play_after=millis()+12000;
      }
    }
  }
  if (cmd) {
    digitalWrite(onboard_led, HIGH);   // turn the LED on to indicate activity
    for (int i = 0; i < retransmit; i++) {
      irsend.sendNEC(cmd, 32);
      delay(70);
    }
#ifdef DEBUG
    Serial.println(cmd, HEX);
#endif
    digitalWrite(onboard_led, LOW);    // turn the LED off - we're done transmitting
  }
}

In other links, How-To: IR Remote Control your Computer