Linux notes

When I recently migrated from KDE4 to KDE5 (yes, Slackware did that update just a few days ago ;-)) I had to discover that no migration was done for my Kopete profile. And not only that, I suddenly had to suffer from various display errors - which had happened a few years ago already, but I forgot the solution to fix this. To make things short: I decided to finally migrate to Pidgin.

Now of course I didn’t want to loose all my chat history, but import it into Pidgin.

Fortunately Kopete is storing the log files as plain XML files, though very strange ones - who thought it was a good idea to safe something like “11 9:5:0” as a time field? Also guessing what that is supposed to say? That’s nine o’ clock and five minutes on the eleventh day of a month. Which month you ask? That’s stored in a dedicated field of course ;-)
Anyway: I’ve just written an XSLT converter again. This time it will also include a small Bash script wrapper for creating the correct directories.

Note that it will only convert Jabber and ICQ out of the box - those were the only protocols I’ve been using with Kopete. It should be trivial to add other protocols though - basically you’ll just have to find out the correct source and target directory names.

The script including a small README file can be found on https://gitlab.com/laenion/kopete2purplelog.

SimCity 3000 was originally ported by Loki Games to Linux, but over time the original binaries became incompatible with modern systems. For that purpose the Loki Compatibility Libraries were introduced, the linked site also contains tons of information to get other Loki to run again. Another option are the Linux Installers for Linux Gamers, which provide updated installers for a lot of old games, and - like in the case of SimCity 3000 - include all the necessary compatibility libraries already.

With SimCity I had one problem additional though. It still wouldn’t start with the following message:

Graphic System: Could not init SDL: No available video device


The problem

After some debugging with strace I noticed that the old version of SDL tries to connect to the local X server via TCP, a feature that was disabled on most distributions in the last few years; if you type ps aux | grep Xorg you will most likely see -nolisten tcp somewhere in the options, which will prevent opening that TCP port.

The solution

If you are using the LIFLG installer from above you can find a very useful helper script that allows you to play the game in a separate X server instead. Just open the file sc3u.sh in your installation directory and uncomment all the XSERVER, XSERVER_OPTIONS and XSERVER_DISPLAY lines, and in the XSERVER_OPTIONS line change -nolisten tcp to -listen tcp. You will probably also have to create a file called /etc/X11/Xwrapper.config with a line allowed_users = anybody to be able to start the X server as a user (see man Xwrapper.config for details).

Otherwise you could explicitly start your X server with -listen tcp instead. How that is done depends on your display manager. For KDM for example you can find the X options in /etc/kde/kdm/kdmrc → [X-:*-Core] → ServerArgsLocal, other display managers should have similar configuration files. This may not be the best option for security reasons though.

Short version:
Just visit the project’s website and follow the 5 simple instructions (sounds like clickbait, doesn’t it? ;-)) in the file README.md.

Motivation

When I accidentally messed up my installation of Sailfish OS on my Gemini phone I had to use Android as a backup until I found the time to fix the issue. In the meantime several SMS arrived, which I obviously also wanted to see in Sailfish OS. So I started to investigate the possible options.

Development

My naive attempts to just export the database from the Android partition failed miserably due to Android’s security mechanisms. In the end I settled by using SMS Backup & Restore to get a backup of my messages. Not only does this app seem to be the de-facto standard for SMS backups, but it also provides good documentation on the format of the generated XML file.

I already used commhistory-tool to import my SMS history from Firefox OS, so commhistory-tool’s JSON format was set as the target format. And with an XML file as the source: What would be suited better for conversion than an XSL Transformations (XSLT)?

Already preinstalled on my (Linux) system is one of the most commonly used XSLT processors - xsltproc, so I decided to use that one. Unfortunately I had to find out this processor only supports XSLT 1.0, so is missing a lot of convenience functions. On the positive side it supports several EXSLT commands instead, e.g. for dates, strings and functions, which are used extensively.

I later found out that using xsltproc also has another advantage: It is also preinstalled on Sailfish OS! This completely eliminates the need for a separate PC to convert the files - you can do that directly on your Jolla / Sailfish OS phone. In the case of the Gemini I was able to just export the file to the memory card in Android, reboot into Sailfish, generate the JSON file and import it. Could this be any easier?

The XSLT file can be found on https://gitlab.com/laenion/android-sms-backup-restore-2-jolla-commhistory-tool/blob/master/android-sms-backuprestore-2-jolla-commhistory-tool.xsl, the project’s README.md file contains detailed instructions on how to use this file.

Introduction

Welcome back to the second part of the Firefox OS migration guide. Last time we migrated the Firefox OS Offline Calendar data, this time we will take a more general approach to export any kind of data, e.g. your SMS history.

Preparations

This article assumes you retrieved your user data from your Firefox OS device already - if you didn’t, please follow the guide in the article Exporting the Firefox OS Offline Calendar (Firefox OS migration part 1) up to step 5 first.

Checking what data is available

In the last article you may have been wondering what that Database name: field is for: It’s for getting the contents of any kind of data. The default value of b2g-calendar will get you the IndexedDB database containing the calendar data.

So let’s check which other databases exist. Unfortunately a website can’t access that data, but Firefox itself can:

1. Open the website http://digitalimagecorp.de/software/firefox-os-data-exporter/ffosexporter.html like in the previous article (if you created a separate profile please use that profile to open the website!)
2. Open the Firefox menu by clicking on the Open menu button at the top right of your browser.
3. Click on Developer → Toggle Tools.
4. A toolbar will open on the bottom of the browser. At the top right of that bar click on the gear icon (Toolbox Options).
5. In the section Default Firefox Developer Tools check Storage.
6. A new tab called Storage has appeared on the top of the toolbar now - click on it.
7. You will find a section Indexed DB on the left. Click on it and also click on the http://digitalimagecorp.de entry.
8. You will get a list of all databases you migrated initially.

If this guide was too confusing see the Mozilla manual on how to open the Storage Inspector.

Importing more data

You have probably backed up your complete device’s user data, so let’s see what else is there. Open the backup with your file manager and navigate to the directory containing several directories named like 1015+f+app+++clock.gaiamobile.org - those are the applications that used IndexedDB databases to store their data. You will also notice the folder called chrome: This one contains the data of the core Firefox OS applications like your SMS storage.

Now repeat Step 5 of the original tutorial, but this time copy the (complete) contents of the idb directories of the applications you are interested in into your newly created http+++digitalimagecorp.de/idb directory. Unfortunately several applications have files of the same name, so you cannot use them at the same time (you have to repeat Step 5 for each one your are interested in).

Working with the data

Now you can just check which databases are available using the Firefox Storage Inspector view from above. If you want to export any of those databases enter the name of the database in the Database name field and click on Export to get the contents as JSON output. There are numerous tools available on the web to transform JSON data into other formats (or you can write one yourself if a converter to your format is not available :-P).

Wait - what about my SMS data?

You will find those in the chrome directory of your backup. When you import those into Firefox you have to export the sms database, the required Object Store is also called sms.

If you own a Jolla phone or any other device based on Mer you can simply select Convert SMS data to commhistory-tools JSON file, save the file to your phone and import it with the command commhistory-tool import-json sms.json (Note: MMS messages are not supported by this tool unfortunately).

If you have an Android phone there seem to be several tools available to import SMS data from CSV files, so one simple solution would be to convert the JSON data into CSV. There are various converters for that available, but http://konklone.io/json/ looks very promising because it also processes all data locally on your PC (i.e. no data is sent to the server as of 2016-03-21).

Fin

This concludes my series of the Firefox OS migration guide. I hope you found it useful, and don’t hesitate to ask if you have any questions. (The Comments section is currently locked due to the massive spam amount, I hope I will be able to unlock it soon again with a better filter.)

Update 2016-03-21: Improved visual appearance and simplified steps 5 onwards - conversions can be done in the browser directly now without having to use Python.

Introduction

This how-to will describe a way to export the calendar entries of your Firefox OS phone and convert the data into a standard iCalendar / ICS file that can be imported into almost any program / phone / online service.

Why is this even necessary? When Mozilla introduced the Firefox OS product line it started with big promises: Just create a HTML5 application, and it will run on every device including Firefox OS. Unfortunately Mozilla forgot a few essential things, for example to include an API to access calendar data. If you have been using the offline calendar (or your CalDAV server went offline) then the calendar ended as a data sink - you can put stuff in, but you won’t get it out any more with conventional means.

When migrating to another phone (or if you want to get a useful backup copy) you may still want to migrate your calendar data. This article will guide you through this (lengthy) process. A second article for the migration of SMS / MMS data is planned.

Requirements

• Mozilla Firefox (tested with version 38esr & 45esr)

This how-to was created on Linux with a German ALCATEL ONE TOUCH FIRE, updated to the official Firefox OS 1.3.0.0 version by the provider. Just saying in case something is working differently in your setup - the instructions should be quite generic and platform independent though ;-)

Step 1: Basic setup

First things first: Both your computer and your phone will have to be configured to allow them to communicate with each other. We will configure the phone, install the Android Debug Bridge extension in Firefox and try the first connection to the phone.

1. On your phone go to Settings → Device / Device information → More Information → Developer settings / Developer and select Remote debugging.
2. In Firefox press the Alt key and select Tools → Web Developer → WebIDE in the menu.
3. In the Firefox WebIDE select Select Runtime → USB DEVICES / Install ADB Helper.
4. Install the ADB Helper Add-on.
5. Choose Select Runtime again and click on your device in the USB DEVICES section.
6. Accept the connection request on your phone.
7. Back in the Firefox WebIDE select Runtime → Runtime Info.
8. Check the value of ADB is root.
9. If it’s no click on root device.
10. Remember if it says yes or no now (you will need it for Step 3).

If any of these steps didn’t work go to Mozilla’s troubleshooting site - you will most probably find the solution to your problem there.

Step 2: Connecting to the device on the command line

Excellent - the connection to the phone is working; unfortunately Firefox’s WebIDE doesn’t support transferring files or booting a custom kernel, so we will have to use ADB on the command line.

1. In Firefox enter about:support into the address line.
2. Select Application Basics / Profile Directory / Open Directory.
3. In your file manager navigate to extensions → adbhelper@mozilla.org.
4. You will find a directory named after your operating system here (e.g. linux, linux64, win32 or mac64).
5. Open this directory in a console (Linux / Thunar: right-click on the directory and choose Open Terminal Here; Linux / Dolphin: right-click on the directory and choose Actions → Open Terminal Here; Windows: How to Use “Open Command Window Here” in Windows 7 (inofficial instructions); Mac OS: Open terminal here in Mac OS finder (inofficial instructions)).
6. Make sure your phone’s display is turned on and the screen is unlocked - otherwise your device won’t be listed in the next step.
7. Type ./adb -l (on Linux and Mac) or adb -l (on Windows).
8. You should see the (internal) name of your device now.

Step 3: Getting root access to your device

Check if your phone is locked by the vendor and (temporarily) boot an alternative kernel if it is.

1. Remember step 1 when we checked if ADB is root said yes or no? If it was yes: everything is fine and you can directly go to step 4.
2. Otherwise you will have to find a kernel allowing root access for your device. You should be able to find one on https://firefoxos.mozilla.community/devices/ - just choose your device and download the appropriate file. Note: We don’t want to flash the whole device, so most of the documentation on the site isn’t necessary.
3. You will probably have downloaded a ZIP file - in my case I’ve downloaded the “Version 1.4” image, as that version was closest to the one installed on my phone; I’ve also tried a 2.6 image successfully, so you will probably be fine with any version you can get (don’t blame me if anything goes wrong though :-)).
4. Extract that ZIP file - it should contain a file called boot.img, which is the one we need.
5. Restart your phone and keep the volume down button pressed. Keep it pressed for five seconds after seeing your device manufacturer’s logo.
6. If the manufacturer’s logo stays there you have now successfully entered fastboot mode.
7. On the command line type ./fastboot boot "<path containing the extracted files>/boot.img" (in my case the patch was /tmp/14_20150406/boot.img; leave out the ./ at the beginning of the line on Windows; on Linux you have to be root for this command to work).
8. Your phone should continue booting as normal.
9. When the boot has finished and you unlocked the screen try connecting to your phone via Firefox WebIDE again - you should get ADB is root: yes now.

Step 4: Getting the data from the phone

Now that everything is working we will transfer the calendar data from the phone and store it on your PC.

1. For the sake of convenience we will do a backup of all your data (otherwise we would have to find out the exact file name for your device first); type ./adb pull /data/local/storage on the command line (again without the leading ./ on Windows).
2. When the transfer is complete a directory called persistent will contain a backup of your data; if you have a very recent version of Firefox OS you may have gotten three directories called default, permanent and temporary instead - this will make a difference in step 5. In any case I’d recommend using your file manager to move those directories out of the Firefox profile now and keep them in a save place ;-)
3. If you want you can reboot your phone now (and get the original kernel back) - we have all the data we need.

Step 5: Importing the data into Firefox

You just backed up several SQLite files.

In case you are interested some technical background: Firefox OS is using IndexedDB to store it’s data. IndexedDB itself is using an SQL database, however the data is not stored in plain text, but only as an internal serialized representation; this is also the reason why we can only use Firefox to decode the data again: Other browsers have different internal representations and wouldn’t be able to decode it. Fortunately the serializer is the same for desktop Firefox and Firefox OS…

The data will be imported into Firefox now. IndexedDB data is always connected to a specific URL, so we will have to create a fake one.

If your phone’s backup data resides in a directory called persistent proceed with step 5a, if you got default and permanent instead proceed with step 5b.

Step 5a: When you got a “persistent” directory

1. You got the data in an old format. In the following steps a new profile will be created, as the databases have to be migrated to Firefox’s new internal format first.
2. Close Firefox.
3. Start Firefox’s ProfileManager by typing firefox --ProfileManager --new-instance on the command line.
4. Click Create Profile….
5. Click Next.
6. Enter a profile name (e.g. Firefox OS import)
7. Remember the directory that is shown in the middle of the dialogue and click Finish.
8. Don’t start Firefox yet!
9. Open the directory you just created in your file manager - it should be almost empty.
10. In the profile directory create a new directory called storage.
11. In the storage directory create another directory called persistent.
12. Inside persistent create another directory called http+++digitalimagecorp.de
13. Inside that directory create a directory called idb.
14. Now copy all the calendar data from your phone backup into this directory. You will need the sqlite and sqlite-journal files - on my device they were located in persistent/1012+f+app+++calendar.gaiamobile.org/idb/.
15. Now you can finally start Firefox - but make sure that Use the selected profile without asking at startup is not selected; Firefox will migrate the data to it’s new storage format, so don’t be surprised when the directories you just created don’t exists any more afterwards.

Step 5b: When you got a “default” directory

1. Switch to your file manager: The default directory of your phone backup data should contain a directory ending with calendar.gaiamobile.org.
2. Copy this directory into the directory storage/default of your Firefox profile.
3. Now rename the directory calendar.gaiamobile.org to http+++digitalimagecorp.de.

Step 5: Continued (both cases)

1. In Firefox open the URL http://digitalimagecorp.de/software/firefox-os-data-exporter/ffosexporter.html. If you just created a new Firefox profile (Step 5a) use the new browser window to open the URL!
2. Click on Export database and press the Export button to get a dump of all related databases.

If the page just says Done without any further output you probably didn’t create the directories correctly - remove the storage directory in the profile and try again…

Step 6: Convert the data into ICS format

Now the only remaining step is to convert the data into something usable by other programs - we will use iCalendar / ICS files for this purpose.

1. The required data can be found in the section events.
2. Just click the button Convert Calendar data to ICS file below that textfield.
3. Now you can save the converted data using the Save button.

Congratulations, you have made it. You should have a new file called “events.ics”, a standard ICS file containing all your Offline Calendar entries, now - use it with any program you like.

FAQ

This sounds complicated - isn’t there any easier way

I’m not aware of any - as said Firefox OS doesn’t provide any API for calendar access. Syncto aimed to provide calendar synchronization among others, but as far as I can see only Bookmarks and History synchronization are implemented. Backup and Restore Firefox OS data (and a few others) are essentially doing the same, but don’t provide a way to export the data (i.e. they are missing steps 5 & 6).

I don’t want to upload my data to you!

You won’t - the whole export / conversion process happens locally on your computer.

I don’t understand how to do xxx

Feel free to use the issues section on GitHub and ask away!

References

The following sites were particularly useful for finding out about Firefox OS’ way of operation:
Hacking FirefoxOS: Excellent information about flashing and building a custom kernel for your Firefox OS phone
Firefox OS Builds, imágenes Vanilla actualizadas para ZTE y Alcatel (Spanish): I found out about http://firefoxosbuilds.org / https://firefoxos.mozilla.community here; links to Firefox OS builds for Alcatel and ZTE phones
Actualización de Firefox OS 1.0 y 1.1 a Firefox OS 1.3 , 1.4 Y 2.0 (Spanish): Another blog with links to Alcatel images
Instalado las roms de Firefox OS (Spanish): And another blog with a detailled flashing guide
iCalendar documentation
IndexedDB: Subdomain Quota Management: Mozilla bug documenting the movement of the IndexedDB directories
idbbrowser: Browse Firefox’s IndexedDB databases
http://b2g.vegnux.org.ve used to provide images for different phones, but unfortunately that site doesn’t exist any more.