Category Archives: Software

04Apr 2013

Bulk Downloading Collections from Archive.org

⋅ 5 Comments

Archive.org is one of my favourite sites on the whole wide interwibble. It’s a massive storage facility – an archive, if you will – for digitised media of all kinds. Books, magazines, music, video – it’s all on there and completely free of charge.

Trouble is, actually getting it isn’t always easy.

Don’t get me wrong: the Archive.org website is great. Clicking on a magazine from 1988 and being able to read it right there in my browser, with page-turning animations and all, is something special – but I want to be able to download the files, stick ‘em on my tablet and read them in the bath.

Thankfully, Archive.org is cognisant of this requirement: the site officially supports downloading in bulk using the excellent wget utility, but the list of instructions isn’t exactly straightforward. According to Archive.org, you need to use the advanced search function to return a list of identifiers in CSV format, then strip out a bunch of formatting and an excess line at the top, then feed the modified file to wget…

Wait, change the formatting of the file? Manually download the CSV through your web browser? Isn’t there a better way?

No, it turns out. Well, no it turned out at the time. Yes, now, because I’ve written one: archivedownload.sh. It’s a hacked-together shell script that chains a few useful GNU tools – wget, sed and tail primarily – and makes it possible to download an entire collection from Archive.org at the command line, no messin’.

All you need is the script – available on GitHub – and the name of the collection you want to download. For argument’s sake, let’s say I want a copy of every issue that Archive.org holds of Acorn Programs Magazine. First, I need the collection name: looking at an individual entry page, I see that’s “acorn-programs” – which I could probably have guessed, to be honest.

At my terminal – at which I’ve downloaded a copy of archivedownload.sh and made it executable – I need only issue the following command:

./archivedownload.sh acorn-programs

Voila: the script grabs the CSV identifier list from Archive.org’s advanced search, formats it in the way wget expects, and then tells wget to go off and download the PDF format files. Due to the way Archive.org is laid out, this takes a while: the CSV doesn’t contain the locations of the files, just their identifier, so wget has to spider its way around the site to find them. For the five issues of Acorn Programs – for which the script downloads ten PDF files, five featuring scanned images and five featuring OCRd reflowable text – wget will actually end up downloading nearly 200 files, most of which it discards. Wasteful, yes, but Archive.org seems to be happy doing it that way.

EDIT: Thanks to Archive.org’s Hank Bromley, the author of the original instructions for bulk downloads, the wget command is now a lot more efficient: instead of downloading 200 files for 10 PDFs, it now downloads 20 files. See this comment for details.

The script downloads the PDF format files by default; you can change this in line 18 to download a different format like ePub if you’d prefer. Just find the section that says “-A .pdf” and change it to “-A .epub” and re-run the script.

Just a few things to note: the script downloads the PDF files to the current directory, and if it finds any text-format PDFs – not all collections have them – moves them into a sub-directory called “text,” which it will create if it doesn’t already exist; also, it’s pretty indiscriminate in what it does, so make sure you run it in a folder that doesn’t have any existing PDFs if you don’t want weird things to happen. If you want the script to continue downloading even after you close your terminal, call it via screen as “screen /directoryname/archivedownload.sh collection-name” then CTRL-A and CTRL-D to detach the session from your controlling terminal. Oh, and if the script gets interrupted part-way through, delete the most recent PDF file – which will likely only be partially complete – and re-run the script: wget is set to no-clobber, so it won’t download any PDF files that have already been downloaded, which is handy if it fails towards the end of a large collection.

Enjoy!

24Mar 2013

Mounting Amiga FFS Hard Drives Under Linux

⋅ 1 Comment
68030

I had cause to need to mount a hard drive – actually, a Compact Flash card, but that’s beside the point – from an Amiga on my PC the other day. In theory, not a problem: Linux includes in-built support for Amiga FastFileSystem (AFFS) devices, so it should just be a case of identifying which of the three partitions I’m after and giving the mount command.

So, let’s fire up fdisk:

$ sudo fdisk -l
 Disk /dev/sdc: 4009 MB, 4009549824 bytes
 124 heads, 62 sectors/track, 1018 cylinders, total 7831152 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disk identifier: 0x00000000
Disk /dev/sdc doesn't contain a valid partition table

Huh. That’s odd – it certainly should. Other partitioning tools say the same thing – and the reason becomes clear: apparently, while Linux technically understands AFFS partitions, it doesn’t actually understand Amiga partition tables. At least, most packages don’t: thankfully, one package does. So, to the wonderful parted:

$ parted /dev/sdc
 WARNING: You are not superuser. Watch out for permissions.
 GNU Parted 2.3
 Using /dev/sdc
 Welcome to GNU Parted! Type 'help' to view a list of commands.
 (parted) u
 Unit? [compact]? b
 (parted) p
 Pralloc = 0, Reserved = 2, blocksize = 1, root block at 62832
 Pralloc = 0, Reserved = 2, blocksize = 1, root block at 2112896
 Pralloc = 0, Reserved = 2, blocksize = 1, root block at 5965912
 Model: Generic STORAGE DEVICE (scsi)
 Disk /dev/sdc: 4009549824B
 Sector size (logical/physical): 512B/512B
 Partition Table: amiga
Number Start End Size File system Name Flags
 1 278528B 64061439B 63782912B affs1 DH0 boot
 2 64061440B 2099544063B 2035482624B affs1 DH1
 3 2099544064B 4009549823B 1910005760B affs1 DH2
(parted) quit

You may notice I told parted to switch unit type from the default – Compact – to Bytes. There’s a reason for that: because the kernel can’t see the partition table, it hasn’t created the usual sdc1, sdc2 and sdc3 devices under /dev – meaning I can’t tell mount where to look for the third partition, which is the one I’m after. A problem – but one that can be resolved by giving mount an offset option, taken from the ‘Start’ column of parted’s output:

$ sudo mkdir /media/AmigaMount
$ sudo mount -t affs -o offset=2099544064 /dev/sdc /media/AmigaMount
 mount: wrong fs type, bad option, bad superblock on /dev/sdc,
 missing codepage or helper program, or other error
 In some cases useful info is found in syslog - try
 dmesg | tail or so

Huh? What does dmesg have to say about this?

$ dmesg | tail -1
 [83740.730900] AFFS: No valid root block on device /dev/sdc

Okay, so that didn’t work. Now it’s time for the brute-force option: if mount’s offset command won’t work, let’s try setting up a loop device on the partition’s offset:

$ sudo losetup -o 2099544064 /dev/loop1 /dev/sdc
$ sudo mount -t affs /dev/loop1 /media/AmigaMount
$ ls /media/AmigaMount
 A500_A600_EtoZGames Disk.info Favorite Games.info
 A500_A600_EtoZGames.info Favorite Games

Bingo! At this point, you can read or write any file you like to the partition. To mount a different partition on the disk, simply give losetup the offset of that partition as reported by parted – remembering to tell parted to switch to bytes as its unit.

Having finished sticking my files on the drive, it’s time to tidy up and unmount:

$ sync
$ sudo umount /media/AmigaMount

Eject the drive, stick it back in the Amiga and lo: my files are there.

An alternative method to the above is to use an Amiga emulator on your PC: UAE, for example, can mount a real Amiga drive. This way’s easier, though – at least, if you don’t have to go through the troubleshooting steps that brought me to my understanding of the flaw.

To recap: insert Amiga drive, use parted in bytes mode to get partition offset, setup loop device at that offset, mount loop device. Not as simple as it should be, but hey: it works. This also works, interestingly enough, on disk images: create a backup of your Amiga drive with dd and you can then mount partitions directly from the backup rather than the real drive.

31Oct 2012

Fixing the Android Camera Date Ordering Bug

⋅ Leave a Comment
ExifTools

I upgraded my Galaxy SII to a custom version of Android 4.0.4 Ice Cream Sandwich the other night, and it’s been brilliant – except for one little bug. If I go into the Gallery app, images on my microSD are in a semi-random order. If I tell Android to group by time, the problem becomes apparent: large numbers of images, always in contiguous blocks, are thought to have been taken between 1992 and 2018. I didn’t have an Android phone in 1992, and 2018 hasn’t happened yet – clearly, it’s a bug.

A quick search turned up this Google Code issue, which matches what I’m seeing exactly: GPS data is sometimes corrupted as it writes to the image. Worryingly, the bug dates back to at least Android 2.3 Gingerbread – which I can confirm, as the images in question were taken while the Galaxy SII was still running 2.3.4 – but Google promises a fix is incoming.

Here’s an example of the problem. The GPS Date tag from a non-corrupt image reads 2012:10:30 – year, month, day with a colon delimiter. The GPS Date tag from a corrupt image reads 0112:09:26 – a clearly invalid date which is confusing the heck out of the Gallery app. (You can check this out yourself with the command exiftool -gps:GPSDateStamp filename.jpg)

But why have I only just spotted the problem? Simple: the ICS – and Jelly Bean – Gallery app has changed its behaviour from previous versions. Where the old Gingerbread Gallery used to use the file modified date to sort the images, the ICS and newer Gallery uses the GPS Date Stamp tag from the EXIF data – the very tag that is being corrupted by the bug. Why it does that, I have no idea: there are numerous other EXIF tags which have the right date stamp and would be more appropriate for sorting, such as the Date Time (Original) tag. Perhaps that’s Google’s fix: modify the Gallery to sort on a different tag.

A pending fix doesn’t solve the fact that I’ve got a few hundred images with corrupt EXIF information, though. Switching to a third-party image viewer, like QuickPic, orders the images correctly, but that’s a workaround at best. So, I’ve written a quick tool to correct the invalid EXIF data on all my images.

It’s a bash shell script, and simple enough: it calls the ExifTool Perl package to copy the date from the Date Time (Original) tag over the top of the GPS Date Stamp tag. The result: the invalid tag goes away, Android knows what year the image was taken, and it orders the gallery correctly. It’s a hack: it overwrites the GPS Date Stamp of all JPG images in the current directory, regardless of whether the year makes sense or not, but it fixes the problem – albeit slowly.

You can find the script on GitHub. Hope it helps!

When you’ve corrected your images, you’ll need to go into Settings and Applications on your Android phone. Choose ‘All,’ find the Gallery app and tap it to get into the menu. Hit ‘Force Stop’ and then ‘Clear Data’ – don’t worry, you won’t lose any images, this just deletes the file database and thumbnail cache. If you don’t, the Gallery will still use the old ordering data and you’ll still have a problem. When that’s done, load the Gallery, let it find your files, and enjoy having all your images in the right order.

Oh, and one other thing: take a backup of your images before running the script. Although it worked fine for me, I can’t guarantee it won’t corrupt images further – and I don’t want to be responsible for you losing your precious photos.

01Sep 2010

Android Rooting: Pre-Installing Apps

⋅ Leave a Comment
Android Terminal Emulator running on a Milestone

Android Terminal Emulator running on a MilestoneRooting your Android smartphone can offer some pretty amazing benefits, from simple things like the ability to take screenshots to the option to use your handset as a Wi-Fi hotspot.  Perhaps the most impressive, however, is the option to change the apps that are pre-installed on your ‘phone, and replace them with your own.

In the case of my handset, a Motorola Milestone, app memory is at a premium: with just 196MiB of user-accessible storage on-board, larger apps can often mean having to remove something that’s already installed.  Thankfully, there’s a cheater’s way of freeing up some much-needed space: turn your optional apps into pre-installed versions.

First things first: you’re going to need a rooted handset.  If it’s not rooted, this tutorial will get you nowhere.  If you’re new to rooting, try the Easy Root Compatibility Test which offers certain ‘phones a one-stop solution.  If your handset isn’t supported, try one of the many guides on-line.

Once you’re rooted, you’re going to need a terminal emulator.  Install that, and you can get access to a shell – which is where the fun begins.

Step one – find out what’s mounted where with the command

 mount

You’ll see that we have a bunch of devices mounted in a bunch of locations, but the one we care about is /dev/mtdblock6, mounted as /system.  It’s in this special directory that Android stores all its internal gubbins – and to stop you from fiddling around with it, Google has very sensibly made it read-only.

Sadly, that also stops us from moving apps from the very limited user memory into the otherwise wasted system memory – and from deleting unwanted pre-installed apps, such as the Motorola-branded sat nav package that comes with the Milestone or the Facebook app that arrived unexpectedly as part of Android 2.1.

So, what can we do?  Easy: we re-mount the device as read-write.  You’ll need to become root using su, after which you can remount the block device.  Just type the following commands in to the terminal:

su -
mount -o remount,rw /dev/mdtblock6 /system

Now we have full read-write access to the /system file system. Bring up a directory listing of /system/app/ with:

ls -l /system/app/

You’ll notice that all the .apk files for the pre-installed apps are here.  Delete any .apk with rm and it’s gone for good – so for goodness’ sake be careful.

We can use the same command to take  a look at the .apk files for the apps we’ve got installed in user memory, too:

ls -l /data/app/

To move a package from user memory to system memory, making it ‘pre-installed’ and freeing up useful user memory, just do the following:

cp -f /data/app/com.publisher.packagename.apk

The file will be automatically removed from user memory and appear to vanish – but when you restart your ‘phone, it will re-appear as a pre-installed ‘system’ application.  Do this with a larger app, and you can free up a lot of valuable space.

A note about updating: because /system is mounted as read-only, the Android Market can’t write to it in order to update packages.  Instead, if an update for a pre-installed package is available, it will write the data back into user memory.  So, if you’ve saved 4MiB by moving Google Maps into /system, you’ll lose it all again the next time Google releases an update.

While it’s possible to get the space back – simply repeat the method of moving the file to /system again – it’s probably better to move applications that don’t get updated all that often.

That’s it: you can now enjoy significantly more space for apps!

30Aug 2010

aRSSduino

⋅ 1 Comment
The aRSSduino in action

The aRSSduino in actionThe aRSSduino is a simple project for the Arduino microcontroller, designed to display RSS feed entries on a 16×2 LCD.  It’s still in the early stages, with the following outstanding:

  • To Do: Support for larger LCDs
  • To Improve: Better UTF-8 character handling
  • To Do: Multiple RSS feed support

For now, however, it’s a pretty neat hack – and an alternative back-end allows it to display Twitter @ replies instead, with the person’s username on the top line of the display and the message below.

The aRSSduino relies on a USB connection between the Arduino and the host PC – it’s not a stand-alone project.  Currently, the Python back-end is written to run on a Linux-based host – although it should be relatively simple to port to Windows, I have no plans to do so at present.

You can download the project source code – both for the Arduino sketch and for the Python-based back-end here.  If you improve upon it, let me know!

03Apr 2010

Recommended: AppBrain

⋅ Leave a Comment

AppBrain screenshotI’ve stumbled across a neat toy in the form of AppBrain, a snazzy bit of kit for Android ‘phones that finally brings the one feature I’ve been missing since getting my Milestone: a desktop interface to the Android Market.

It’s a simple concept: you install a client app on your ‘phone, and then it syncs to the AppBrain website – free of charge, of course – and allows you to browse your installed applications.  Where things get clever is that you can queue up tasks – such as uninstalling an application or installing a new one from the Android Market, which is fully accessible from AppBrain’s site – and then sync them to the client app, which carries out your demands.

It’s a neat solution to a thorny problem – finding decent applications while using the quite frankly cramped Android Market interface on the handset itself – and the fact that it’s free is the icing on the cake.

Oh, and it allows you to share your installed apps too – like so:

Although as you can possibly see, it has a tendency to limit itself to the first few in the list.  Still, if you want to install it and give it a go yourself just scan the QR Code below.

QR Code for AppBrain

02Apr 2010

iPad killer app: comics?

⋅ 4 Comments

Marvel Comics for iPadI’ve been scoffing at the iPad since its details were first announced: no Flash support; no multitasking; no third-party unapproved apps; why would I ever want one?

Well, I think I’ve just found the answer in the Marvel Comics app.  Quite frankly, it looks awesome.

The guys over at Boing Boing have got the details, and the included video makes it look like the sort of thing with which I, quite frankly, need to be involved.

I’ve been reading comics – well, mostly manga – on my Sony PRS-505 for a while, converting them to PDF and futzing with the images to better suit the 8-colour greyscale eInk screen, and it’s got me hooked on digital comics.  While that single app certainly wouldn’t make me rush out and buy an iPad – hell, $500 gets you a hell of a lot of paper comics – it’s definitely making me more interested, and I’ll be keeping an eye out for the inevitable Android- and Ubuntu-based slates as they get released.

Well, that or I’ll finally get around to converting my old Eee PC into a touch-screen enabled tablet…

25Mar 2010

Firefox issues on Ubuntu 10.04

⋅ 1 Comment

Having finished my articles this morning, I figured it was time for an apt-get dist-upgrade on the laptop to ensure I was running the latest versions of everything.  A quick download and install later, and it was time to reboot.

At which point Firefox broke completely. No starty, no error – even when launched from the shell.

It’s taken me a little while, but I’ve tracked down the culprit: the Prism plugin.  If you’ve got Prism – a stand-alone Mozilla-based browser for creating desktop-style access to web apps – then the chances are you’ll be bitten by this bug, as installing Prism automatically installs the plugin into Firefox.

Thankfully, the fix is simple: go to Tools -> Add-Ons and disable Prism.

What’s that?  You can’t load Firefox in order to disable the Prism plugin?  There’s a fix for that, too.  Load a shell (Applications -> Accessories -> Terminal) and type:

cd ~/.mozilla/firefox
ls

That’ll give you a listing of the directory – with a single entry, which is your profile folder.  Use cd to change to this directory, then type:

rm compatibility.ini

Once you’ve done that, Firefox should start fine and you’ll be able to disable the add-on.

If you’re curious, there’s a bug filed on Launchpad over here.

17Nov 2009

Gnome Power Management glitch

⋅ 1 Comment

I appear to have broken my netbook.

Well, ‘broken’ might be a bit steep – it no longer responds to a lid close event with the nice, neat standby mode it once treated me to.  Instead, it triggers the standby script and gets itself into a half-on, half-off state.

In this state, the power light is flashing to indicate that it’s in standby.  Unfortunately, it isn’t – everything’s still working fine.  The only indication that it even tried to standby is that my SD card unmount/remount script is triggered and the default keyring is locked.

Read More →

21Oct 2009

Router fun and games

⋅ Leave a Comment

I came home tonight to find that my Internet connection had crapped out, but that’s not what this post is about.  This post is about how Billion – that is, the company rather than the oft-misused numerical value – software engineers are perhaps not the sharpest tools in the box.

After restarting my router, the syslog spat out the following:

Jan  1 00:01:33 DDNS: DynDNS can not be reliable if SNTP -time server do not
reply to modem correctly, Please fix SNTP -server address. 
Oct 21 19:13:50 syslog: NTP current time is Wed Oct 21 19:13:50 2009

That’s the DDNS service complaining that things might go wrong if it doesn’t know the current time – followed by the NTP service updating to the current time.  Apparently making those two things run the other way around is too logical.