Backup and restore a stock Android device

Overview

Taking regular backups of important devices is sensible advice. And backups themselves are no good unless you can restore them. This guide gives a brief overview on how to backup and restore a stock Android device – that is, a device that runs an uncustomised version of Android, that is unrooted. It may also work for non-stock Android devices.

One big caveat: While this is the best method I have found without root, I have not had consistent results across devices when restoring backups, even among Nexus devices. So I recommend that this method is not relied upon exclusively for backup.

Usage information

We will be using adb backup and adb restore to backup and restore device state.

To find out usage information for your device, connect it and run:

adb shell bu help

This info was found by simply running adb.

This guide assumes the backup and restore commands work like stock Android devices.

Backup

  1. Set up your computer and device so that your computer can access the device via USB. There are plenty of good guides out there, but in summary this’ll involve enabling USB debugging on the device and installing the Android SDK Platform Tools.

  2. Run the following command, authorising your computer to access the device if you have not already:

    adb backup -apk -obb -shared -all -system
    
  3. Follow the instructions. In my case, I needed to unlock the device and confirm the backup operation (optionally entering a password to encrypt the backup).

This will store a file in the current working directory.

Restore

  1. Ensure the device to restore to is connected.

  2. Run:

    adb restore <path_to_backup>
    

Using .netrc with curl for Automatic Authorisation

The ~/.netrc file can store credentials. This can include things like OAuth tokens and plain text passwords.

Warning: Be very careful if you decide to store passwords in this file. The file is not encrypted, so you’ll be storing your password in plain text on the disk. This is generally a bad idea!

First, create the file and make it readable by only yourself:

touch ~/.netrc
chmod go= ~/.netrc

Then edit the file to contain lines that look like the following (one line per machine):

machine the.address.tld login the-username password the-password

curl can then read this file by providing the --netrc option:

curl --netrc http://the.address.tld/classified

Ignore a Package Update When Using Fedora and KDE

Recently I wrote about Escaping a login loop when using Fedora. Well, once I had downgraded the package, I wanted to ensure that it would not get automatically updated again!

To ignore the package when performing updated via dnf, add an exclude=<package> line to /etc/dnf/dnf.conf. For example:

[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
exclude=sddm

dnf no longer requests me to update SDDM.

Unfortunately, the built-in updater for KDE (which I blieve is PackageKit) does not honour the settings in dnf.conf. It will, however, honour exclude= lines in the repo files in /etc/yum.repos.d.

Since I wanted to block an update to SDDM, I edited /etc/yum.repos.d/fedora-updates.repo and added an exclude=sddm directive to the [updates] block:

[updates]
name=Fedora $releasever - $basearch - Updates
failovermethod=priority
#baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/$releasever/$basearch/
metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch
enabled=1
repo_gpgcheck=0
type=rpm
gpgcheck=1
metadata_expire=6h
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False
exclude=sddm

After a while, KDE stopped pestering me to install the SDDM update.

I technically probably only have to add the exclude to the repo files, but this is the process that I went through. If you’re feeling adventurous, you may try only updating the repo files.

Escape Login Loop in Fedora

The problem seems to be caused by an update to SDDM (Simple Desktop Display Manager).

On Fedora it can be downgraded, but first you need an internet connection. I enabled my wireless internet connection by running:

nmcli connection up <wireless-network>

Where <wireless-network> is the SSID of my wireless network.

I then downgraded SDDM:

sudo dnf downgrade sddm

Logins should then work after a reboot.

Make kdesu use sudo (a.k.a. stop asking for root's password)

When I tried to start partitionmanager (KDE Partition Manager) on Fedora 26, it would always ask for the root user’s password, despite the fact that the root user was disabled in my setup.

The dialog box that popped up looked like it might be kdesu. kdesu,. a.k.a. KDE su, is a KDE version of su that allows a process to be run as another user, like root. When I tried running kdesu separately it asked for root’s password again. So I knew the problem lay within kdesu, not partitionmanager.

The ever-helpful Arch wiki contained just the golden nugget I was after: a command that would configure kdesu to use sudo:

kwriteconfig5 --file kdesurc --group super-user-command --key super-user-command sudo

This fixed the problem for me, as sudo requests your password, not root’s. Whether you’re allowed to run the command depends on the contents of the system’s sudoers file. Look that file up if you continue to have difficulty.

Re-use Commit Message in Git

To re-use a commit’s message verbatim, including authorship and timestamp information, use -C <commit>:

git commit --amend -C @

Pro Tip: @ is a synonym for HEAD, the currently checked-out commit.

To reset the date as well, use --date now; to reset the author too, use --reset-author:

git commit --amend -C @ --date now
git commit --amend -C @ --reset-author
git commit --amend -C @ --date now --reset-author

If you want to use an existing commit’s message and also want a chance to edit it, use -c <commit>:

git commit --amend -c @

-c is compatible with --date and --reset-author too:

git commit --amend -c @ --date now
git commit --amend -c @ --reset-author
git commit --amend -c @ --date now --reset-author

Uploading to Google Play Music without matching

I’ve had some trouble with Google Play Music’s scan and match service. It sometimes matches tracks to the wrong versions of them. Sometimes the volume of a track is wildly different from what I have locally (with or without ReplayGain). Unfortunately, there is no way to disable matching in Google’s official Music Manager application.

gmusicapi supports uploading to Google Play Music without their “scan and match” service. gmusicapi-scripts utilises gmusicapi. Let’s use them!

  1. Install gmusicapi-scripts:

    # Windows
    py -3 -m pip install gmusicapi-scripts
    # Others
    pip install gmusicapi-scripts
  2. Convert tracks (if needed) to 320kbps MP3 and place them in <directory>.

    • If you place non-MP3 files in this directory, gmsync will try to convrt them with FFMPEG or AVConv if they are on the PATH. Look at the docs to see what formats it supports.
  3. Run gmsync. Follow any instructions it provides:

    gmsync up <directory>

gmsync uploads without matching by default. Perfect.

Working around Intel CPU Running at a Maximum of 49%/50%

My laptop had been feeling slower than it had used to, and I was getting more and more certain that it wasn’t just me.

One time I opened Task Manager and noticed that whenever I put load on the machine, it maxed out at 49%. Every time. Cooling wasn’t the issue as the laptop fan (which is a dumb fan controlled by hardware, not software) wasn’t spinning up much.

After searching online, I stumbled across a thread that would provide the (really rather insane) workaround: Move the Intel processor driver so Windows does not use it. Thanks to everyone there for the discussion.

  1. Restart in the advanced command prompt mode. (Hold Shift when clicking Restart)
  2. Log in to your account.
  3. Run the following:

    cd drivers
    move intelppm.sys intelppm.sys.old
    exit
  4. Select the option to exit and continue to Windows 10.

One comment on that thread suggests that this also works for AMD CPUs by moving amdppm.sys instead of intelppm.sys.

Update 2017-06-23: I previously suggested to add the .bak extension to the driver file. However, Windows recognises this extension, and so it will restore the driver back to its original location at the next restart, meaning the problem will return! I’ve changed the instructions to suggest using .old instead, which I don’t believe Windows recognises, but is still fairly obvious to users.

Tips for Ripping Blu-Ray Audio

I recently ripped my first Blu-Ray audio discs. Here are a few tips I’ve discovered along the way:

tl;dr

Backup the disc using MakeMKV and extract the audio using the updated version of the command under Putting it all together.

Ripping the discs

MakeMKV can be used to rip and, if necessary, decrypt Blu-Ray audio discs. While it is in beta, it can be evaluated by using the current beta key.

I use the complete Blu-Ray backup feature so I have the raw stream files to work with.

Extracting the audio from the video files

ffmpeg can be used to extract the audio track(s) from the video streams on the Blu-Ray.

First interrogate one of your streams using ffprobe to see how the different tracks are laid out:

ffprobe -hide_banner BDMV/STREAM/00002.m2ts

I use -hide_banner to reduce the amount of text printed to the terminal. If you have troubles and decide to ask others for help, be sure to omit -hide_banner as the information it emits can be useful when debugging!

Here is the output I got:

Input #0, mpegts, from 'BDMV/STREAM/00002.m2ts':
  Duration: 00:01:47.07, start: 4200.000000, bitrate: 7133 kb/s
  Program 1
    Stream #0:0[0x1011]: Video: h264 (High) (HDMV / 0x564D4448), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn, 47.95 tbc
    Stream #0:1[0x1100]: Audio: pcm_bluray (HDMV / 0x564D4448), 48000 Hz, stereo, s16, 1536 kb/s

In my case I only have one audio stream to worry about at address 0:1. Note it is labelled s16 which means that it is a signed 16-bit depth audio stream. These details will be important in a moment.

Next comes the extraction:

Update 2017-09-10: I now think it is much more straightforward to trust FFMPEG to do the right thing:

for i in BDMV/STREAM/*.m2ts; do ffmpeg -hide_banner -i "$i" -map 0:1 "$(basename "${i%.*}").flac"; done

Note that I do not set the audio codec, and merely set the file extension to .flac as that is the format that I want.

For reference, this is original command I used:

for i in BDMV/STREAM/*.m2ts; do ffmpeg -hide_banner -i "$i" -map 0:1 -acodec pcm_s16le "$(basename "${i%.*}").wav"; done

The -map 0:1 refers to our address from earlier. The s16 part of -acodec pcm_s16le matches the description of the stream ffprobe gave us. We cannot use -acodec copy as ffmpeg cannot encode pcm_bluray.

Make sure to use .wav at the end of the filename, as ffmpeg considers it important. I was originally trying to use .pcm, but all I got was this error message:

[NULL @ 000000000283dc20] Unable to find a suitable output format for '00002.pcm'

If you have multiple streams, re-run this command, changing -map, -acodec and the output filename (to not overwrite your previous extractions!) as appropriate.

Removing initial silence

After I had extracted the audio, I noticed that every track had about 1.0-1.6 seconds of silence at the beginning. I had no interest in keeping this, so I used ffmpeg with the silenceremove filter:

mkdir no-silence
for i in *.wav; do ffmpeg -i "$i" -af silenceremove=start_periods=1:detection=peak "no-silence/$(basename "$i")"; done

start_periods=1 means remove one block of silence from the beginning of the track, a.k.a. the silence before any audio begins. I use detection=peak because I know I’m working with audio with digital silence; if you’re working with audio that was originally recorded from analogue, omit this option.

Putting it all together

Incidentally, this filter could have been added to the original extraction command, if I knew that I wanted it at the time. :)

Update 2017-09-10: As above, I now think it is much more straightforward to trust FFMPEG to do the right thing:

for i in BDMV/STREAM/*.m2ts; do ffmpeg -hide_banner -i "$i" -map 0:1 -af silenceremove=start_periods=1:detection=peak "$(basename "${i%.*}").flac"; done

For reference, this is original command I used:

for i in BDMV/STREAM/*.m2ts; do ffmpeg -hide_banner -i "$i" -map 0:1 -acodec pcm_s16le -af silenceremove=start_periods=1:detection=peak "$(basename "${i%.*}").wav"; done

Update 2017-05-14: Changed ${i/.*} to ${i%.*} to remove file extensions, as the new way will only remove everything following the last . in the filename, not the first. Thanks to my good friend Mark Holland-Avery for the suggestion!