24 June 2017

Raspberry Pi problems

In early May this year, I set up a Raspberry Pi 3 Wi-Fi access point to cover our backyard. The Wi-Fi of our router upstairs just didn't cut it. The connection to the backyard suffered from catastrophic multipath fading - I could move my device a dozen centimetres to the right and the signal would go from full strength to none; a dozen centimetres more and it was at full strength again. I had never experienced this phenomenon so clearly before and it was quite fascinating (but, as you can imagine, also extremely frustrating to work with).

I used the latest Raspbian at that time and the following tutorial which I highly recommend: USING YOUR NEW RASPBERRY PI 3 AS A WIFI ACCESS POINT. This setup worked quite well, but there was the problem that I could not connect to other PCs on my LAN network when I was connected to my access point. However, that was quickly resolved by commenting out the line with bogus-priv in /etc/dnsmasq.conf. This setting blocks any requests to the private address range, which makes a lot of sense for routers, but none whatsoever for a local access point. I also had the problem that Windows PCs would not find each other in Explorer. I did not have time to resolve that, but I simply used plain IPs to connect to other computers in our network. It feels good to sit outside in the sun with a small laptop while using a remote desktop connection to the big desktop PC inside. However, the remote desktop connection caused a host of other problems because my laptop uses different DPI settings than my desktop so it persistently rearranged all my desktop icons and made the Skype window very blurry (what a 90's problem to have in 2017). It also made the driver support for OpenGL 3 and higher disappear during the the remote desktop session and prevented me from working on my game engine, but that is a matter for another blog post.

Roughly a week later, as I opened up my laptop, I could no longer connect to my Raspberry Pi access point. The process of establishing a Wi-Fi connection would just freeze before completion. It did not indicate a failure, but it was obvious that it didn't work. What is happening? Back to the desktop PC to find out! I logged into the access point using SSH and was greeted with very strange messages:

login as: pi pi@192.168.1.115's password: The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Thu May 11 19:03:13 2017 from 192.168.1.107 -bash: /etc/bash_completion.d/insserv: line 8: syntax error near unexpected token `z' -bash: /etc/bash_completion.d/insserv: line 8: `z' -bash: /etc/bash_completion.d/whiptail: line 2: unexpected EOF while looking for matching ``' -bash: /etc/bash_completion.d/whiptail: line 7: syntax error: unexpected end of file -bash: /etc/bash_completion.d/insserv: line 8: syntax error near unexpected token `z' -bash: /etc/bash_completion.d/insserv: line 8: `z' -bash: /etc/bash_completion.d/whiptail: line 2: unexpected EOF while looking for matching ``' -bash: /etc/bash_completion.d/whiptail: line 7: syntax error: unexpected end of file pi@raspberrypi:~ $ -bash: /etc/bash_completion.d/insserv: line 8: syntax error near unexpected token `z' > -bash: /etc/bash_completion.d/insserv: line 8: syntax error near unexpected token `z' > ^C pi@raspberrypi:~ $

Wait a minute, I never even touched those files. I hadn't even logged in since initially setting it up, nor were any updates installed during that time. I got curious and opened one of the offending files.

pi@raspberrypi:~ $ pi@raspberrypi:~ $ cd /etc/ pi@raspberrypi:/etc $ cd bash_completion.d/ pi@raspberrypi:/etc/bash_completion.d $ ls debconf dphys-swapfile initramfs-tools insserv whiptail pi@raspberrypi:/etc/bash_completion.d $ nano whiptail Error in /etc/nanorc on line 239: Error reading /usr/share/nano/c.nanorc: Structure needs cleaning Error in /etc/nanorc on line 287: Error reading /usr/share/nano/awk.nanorc: Structure needs cleaning Error in /etc/nanorc on line 290: Error reading /usr/share/nano/asm.nanorc: Structure needs cleaning Press Enter to continue starting nano. pi@raspberrypi:/etc/bash_completion.d $

Something's wrong here. There is a problem with the nano configs too? Let's open another file:

pi@raspberrypi:/etc/bash_completion.d $ nano insserv Error in /etc/nanorc on line 239: Error reading /usr/share/nano/c.nanorc: Structure needs cleaning Error in /etc/nanorc on line 287: Error reading /usr/share/nano/awk.nanorc: Structure needs cleaning Error in /etc/nanorc on line 290: Error reading /usr/share/nano/asm.nanorc: Structure needs cleaning Press Enter to continue starting nano. GNU nano 2.2.6 File: insserv # insserv(8) com`letion # # Cmpyriwlt (c) 2009 Kel Moddevman # have insserv && _insserv() z ` local cur prev sysvdi2 services optkons^N cur=`_get_cword` prev=${COMP_WORDS[COMP_CWORD-1]} [ -d /e�c/rc.d/init.d ] && syrvdir=/etc/rc.d/init.d \ || sysveir=/etc/init�d services=( $(echo $systdirm!(README*|*.dpk�*|*.rpm@(orig|new|save))) ) ! services=( ${services[@]#$sysvdir/} ) ( options=( -c --config -d �f -n -o --override -p --path -r -v ) [ Read 34 lines (Warning: No write permission) ] ^G Get Help ^O WriteOut ^R Read File ^Y Prev Page ^K Cut Text ^C Cur Pos ^X Exit ^J Justify ^W Where Is ^V Next Page ^U UnCut Text^T To Spell

The file is completely mangled! Let us inspect some of the mistakes. For example, completion became com`letion. Let's get the ASCII table and look at this. 'p' = 0x70 and '`' = 0x60. Transform that to binary and you get 'p' = 1110000 and '`' = 1100000. Looks like the third bit of this character has been flipped to 0! Similarly, we see that 'Copyright' became 'Cmpyriwlt', that means:

'o' (1101111) -> 'm' (1101101) 'g' (1100111) -> 'w' (1110111) 'h' (1101000) -> 'l' (1101100)

In every case here, one bit has been flipped. Textbook memory corruption. I was using a Kingston 8GB microSD card that approached its 6th year. Even though I didn't write that much to it over the years, it did not age well. Luckily, just days before, I bought another spare microSD card. In 30 minutes, I had the exact same setup up and running again, this time on a new and working microSD card, and it has been working perfectly since.

Addendum and fun fact: I had trouble inserting this blog post into the MySQL table of this website, getting the error

MySQL said: Documentation #1366 - Incorrect string value: '\xEF\xBF\xBDc/r...' for column 'post_content' at row 1

Well... so much for copy-pasting mangled, corrupted files from the SSH sessions into my text document. That moment when you have to open your text file with a hex editor to see where the corruptions are. I just replaced them with question marks.

Addendum on 23 July 2019: I fixed the encoding of the SQL table rows by applying the collation utf8mb4, which is fully compliant with the Unicode Standard UTF-8 encoding. I then pasted the original, corrupted file into the blog post again such that what I formerly replaced with question marks now displays as Unicode replacement characters to highlight the file corruption.