Reliable, Deliverable, Self-Hosted Email

Thu, 08 Jul 2021 11:56:19 +0000

I have been on an ongoing quest to free myself from cloud services for years now. During this time, I have hosted my personal email (@bloomqu.ist) on a ~~Google Apps~~ ~~G Suite~~ Google Workspace account, which, while convenient, also means that my personal emails are at the whims of one of the world’s most privacy-hostile companies.

Google’s famous slogan.

Obviously, this cannot stand forever. I wanted to self-host my email, but I know that self-hosting email is fraught with issues. The problems I wanted to avoid are as follows:

Open-source email software is complicated to set up if you were not a sysadmin in the 90’s.
Popular email providers like Google Mail frequently block emails from residential/public cloud IP addresses for anti-spam reasons.
If your self-hosted mail server goes down, you can potentially miss out on important email.

I cannot use an email service that does not have reliable sending and receiving. Happily, each of these problems has a solution:

The Mailu project bundles antispam, POP3, IMAP, SMTP, webmail, administrative interface, etc. into a set of Docker containers that can be managed with Docker Compose or your container tool of choice. This gives us a good, well-architected base setup to configure further.
Use a trusted SMTP outgoing relay to send email. Yes, this is not self-hosting, but you cannot “self-host” an outgoing email anyways, it is ultimately leaving your network one way or another.
Use a backup email server to receive email when your server is down. Set up MX records with a lower priority than your self-hosted mail server to have them act as a fallback. You can either self-host this backup or use a public service. Then, when the primary server is back up, use IMAP to mirror messages from the backup automatically.

In fact, Mailu is so easy to use, that we can configure (2) and (3) out of the box. Here is a step-by-step guide to configuring Mailu with an outbound SMTP gateway and backup MX server:

Part 1: Install Mailu

Follow the Mailu setup guide published on their website. Make sure to enable the fetchmail service - fetchmail will pull email from our backup mail server.

Set up your DNS records appropriately. For most use cases, this means pointing an A record from your mail hostname (in my case, mail.chary.us) to your mail server’s public IP, and configuring MX records on the domains that will be receiving email:

~ dig A mail.chary.us
mail.chary.us.      300	IN	A	107.191.100.174
~ dig MX chary.us
chary.us.           300	IN	MX	1 mail.chary.us.
~ dig MX bloomqu.ist
bloomqu.ist.        300	IN	MX	1 mail.chary.us.

We do not need to set up SPF/DKIM DNS records at this time, since we will be using an outgoing SMTP relay to send email, not the mail server itself.

Part 2: Set up an outgoing SMTP relay

To prevent other email servers from discarding our emails because they’re from an IP with no reputation for mail sending, we can use an outgoing SMTP relay instead of sending email ourselves.

There are many options available for this, with Mailgun, Mandrill by Mailchimp and SendGrid by Twilio being the three heavy-weights of the industry. I chose to go with SendGrid, because their free plan offers sending 100 emails/day forever, which is far more emails than I can see myself sending.

Once you have an account with your SMTP relay provider, you will want to add the domain(s) that you will be sending from to your account. In SendGrid, this is called “Sender Authentication”. As part of this, you will configure DNS CNAME records for DKIM and SPF:

Confused about how SPF can work like this, even though it’s being set on em0000.yourdomain.com, not yourdomain.com? SPF is used to validate the Return-Path header, not From, and SendGrid uses em0000.yourdomain.com as the Return-Path. 💥. Read more on StackOverflow.

Now that the domains are configured, obtain the credentials for your provider’s SMTP gateway. In SendGrid, this is under the “API Keys” settings panel. Your API key is your password.

In your mailu.env file, update the following environment variables:

# Change to your SMTP relay host - port 587 will automatically use SSL
RELAYHOST=smtp.sendgrid.net:587
# Change to your username, on SendGrid this is the literal string "apikey"
RELAYUSER=apikey
# Change to your password, on SendGrid this is the API key you obtained
RELAYPASSWORD=your-api-key-goes-here

If you have already started Mailu, run docker-compose down; docker-compose up -d to reload the env file.

You should now be able to send an outgoing email (for example, via the Mailu webmail) and by inspecting the Received header at the recipient end, see that it was relayed through SendGrid:

// more headers
Received: from xtranbvx.outbound-mail.sendgrid.net (xtranbvx.outbound-mail.sendgrid.net. [167.12.12.138])
// more headers

If this does not work, you can tail the logs using docker-compose logs -f --tail=0 and re-send the email to see what the error was.

Part 3: Set up the backup email server

If, heaven forbid, your power goes out, your server gets knocked off the shelf, or the fiber line to your house is cut, you will not want to miss out on incoming e-mails. Although some mail transfer agents (MTAs) will retry delivery if the recipient SMTP server is unavailable, you really really don’t want to be relying on “maybe the sender will retry” when it comes to important email.

Luckily, this is pretty easy to fix, by the nature of how MX (Mail EXchanger) DNS records work. Each record has a priority, with the lowest priority records being considered first. Records with equal priority are “randomly” chosen by the sending MTA. For example, with the following MX records, mail would always be delivered to mail.foo.net, unless mail.foo.net is down, in which case it will go to backup-mail.foo.net (10 < 20):

foo.net.   300  IN  MX  10 mail.foo.net.
foo.net.   300  IN  MX  20 backup-mail.foo.net.

You can either set up your own mail server for this purpose, or use a public Internet service as the backup.

If you decide to self-host a backup mail server, make sure the infrastructure is separate from your primary mail server. There is some discussion on the Mailu repo about setting up a backup MX server, and if you DuckDuckGo “backup MX server”, you can find plenty of information about setting up a postfix server in this manner.

I went with the non-self-hosted option - since I already have GSuite set up to receive emails for all my domains, I can use it as the “backup” MX. Here is what my MX records look like:

➜  ~ dig MX bloomqu.ist
bloomqu.ist.		300	IN	MX	1 mail.chary.us.
bloomqu.ist.		300	IN	MX	3 aspmx.l.google.com.
bloomqu.ist.		300	IN	MX	5 alt1.aspmx.l.google.com.
bloomqu.ist.		300	IN	MX	5 alt2.aspmx.l.google.com.
bloomqu.ist.		300	IN	MX	10 aspmx2.googlemail.com.
bloomqu.ist.		300	IN	MX	10 aspmx3.googlemail.com.

As a result, MTAs will first attempt to deliver to mail.chary.us. If my server is down, delivery will fall back to Google Mail.

There are also paid services that offer to do specifically this - a cursory search found this affordable offering.

Once your backup MX records are in place, the last step is to configure Mailu to automatically retrieve email from your backup MX using fetchmail. This is available via the web administration interface. Navigate to /admin and click on “Fetched accounts”. From here, you can configure automatic fetching via IMAP or POP3. My setup for Google Apps looks like this:

Note that Mailu is configured to only pull unread email via “Fetched accounts”.

By default, fetchmail will run every 60 seconds to pull in email.

Part 4: Next Steps

Now you are sending and receiving email in a safe, reliable way, while still maintaining control of your data at rest. 🎉🎉🎉!

Life is happy!

After setting everything up, you may wish to take some additional steps to enhance and secure your setup:

Set up a catch-all address for your domains using Aliases in the Mailu admin panel.
Establish a backup strategy - as with anything self-hosted, the data is now your ultimate responsibility. Nobody will be around to help you recover lost emails when your server’s hard drive inevitably crashes, so establish a strategy now.
Increase your mailbox quota - by default, Mailu creates all accounts with a 1GB mail quota, which is pretty small.
Disable any services you don’t need running 24/7 - this could include the admin panel and webmail.
Test your email setup by sending and receiving emails from your friends. This will help catch any errors with your setup before they manifest into truly embarrassing email problems.

Replacing my phone's battery with a cheap AliExpress knock-off

Mon, 09 Nov 2020 17:06:48 +0000

This is a story of one man’s quest for power.

I purchased my current phone, a OnePlus 5T, in 2017. This summer, after about two and a half years of ownership, I noticed that it was no longer holding a charge all day. Frequently, the phone would reach 0% and shut off, right in the middle of tracking an evening bike ride or watching Netflix while cooking dinner. Although cell phone battery wear is a well-known issue, I got tired of it pretty quickly.

I used the AccuBattery app for about two months to try and get a handle on my battery health. It measured my phone’s amperage draw during the day and used that to estimate that of the 3300 milliamp-hour (mAh) capacity that my battery originally offered, only about 2400 mAh of capacity remained - only about 75% of the battery’s original health:

This answered the question of “why does it feel like my phone is shutting off so quickly?” pretty clearly. Now, it was up to me to get a replacement battery.

Attempting to get a genuine battery

My first thought was that I could simply order the OEM OnePlus 5T battery somewhere online. Why not? I found a page on the OnePlus website where prices are listed for replacement parts. The USA page was down at the time of this writing, but the India support page lists a OnePlus 5T OEM battery replacement as being about $15.

This seemed acceptable to me. My first thought was to email OnePlus support asking how to purchase the battery. Unfortunately, according to the service rep, they do not ship or sell OEM batteries without service:

We would like to inform you that we do not ship or sell the accessories in any parts of the world, and all the repairs are carried out by our Authorized service centers only. So if you wish to get the device repaired, you can send it to the OnePlus authorized service center and get the same repaired.

This is in line with what other OnePlus customers have reported - nobody, as far as I can tell, has ever been able to source OEM batteries from OnePlus, leaving DIY customers like myself to try and find knock-offs elsewhere.

I would’ve sent my phone in for repairs, but after I received the above email, I had such a long and terrible customer support experience trying to arrange the repair that by the end of it, I no longer trusted OnePlus to reliably service and return my phone. This lack of trust was reinforced by horror stories from other OnePlus customers - one customer’s phone was lost by the Fort Worth, TX service center, another’s was lost and took 4 weeks before being returned, and yet another customer had their phone held hostage unless they agreed to repairing EVERYTHING instead of just getting the battery replaced. These stories, combined with my awful customer support experience, convinced me that sending my phone in would be a truly bad idea.

Buying an aftermarket battery

Many people on the /r/oneplus5t subreddit have recommended purchasing a replacement battery from iFixit, but I felt like iFixit was simply selling cheap Chinese batteries with a nice label on them. I mean, if OnePlus can fix it for $15, why does the iFixit battery cost $30, if not for marketing?

So, I hit up eBay and AliExpress, and eventually found the [sic] “Specail Mobilephone Parts Store”, where they offer a “4650 mAh” “Perfect business battery” for the OnePlus 5T. With slogans like “Giant energy; huge capacity”, “Safety does not explode”, and “Ensure qualified and safe to use”, I felt confident that my $11.87 was going to a good place. I placed the order and, about 3 weeks later, I received the battery in my mailbox.

Battery Physics 101

The OEM battery (left) and the aftermarket battery installed (right).

The first thing I noticed about the replacement battery was that the capacity was even HIGHER than what I ordered. The OnePlus 5T OEM battery is rated at 3300 mAh capacity, the AliExpress product page advertised a battery with 4650 mAh capacity, and the label on the battery I received claimed an astounding 5350 mAh capacity - 162% of the OEM capacity. Clearly, I had gotten a great deal!

The second thing I noticed was that the aftermarket battery was significantly lighter than the OEM battery. So much lighter that I weighed the batteries out of curiosity. The OEM OnePlus 5T battery weighed 47.0g. The aftermarket OnePlus 5T battery weighed 38.7g, or about 17% less.

It’s amazing that Da Da Xiong was able to achieve 162% capacity with 17% less weight. Too amazing to be true, in fact.

Via Wikipedia, I learned that the specific energy of a lithium-ion polymer battery can be up to 265 watt-hours per kilogram (Wh/kg). The nominal voltage of the lithium-ion polymer batteries here is about 3.8V. We can use Ohm’s law to calculate the maximum possible capacity of each battery based on weight, assuming that each battery is always supplying the nominal 3.8V.

Let’s start by calculating the maximum possible Amp-hours (Ah) per kilogram (kg) for a Li-ion poly battery at 3.8V, using Ohm’s law:

265 Wh/kg / 3.8 V = 69 Ah/kg

Now, we can calculate the maximum physically possible capacity for each battery by multiplying this number by the weights of each battery:

OEM battery:          .047 kg * 69 Ah/kg = 3.2 Ah = 3200 mAh
Aftermarket battery: .0387 kg * 69 Ah/kg = 2.6 Ah = 2600 mAh

The astute reader might be wondering why this estimate for the maximum capacity of the OEM battery (3200 mAh) is less than the capacity OnePlus advertises (3300 mAh). Why is this? Well, it’s because the assumption we made - that each battery is always supplying the nominal 3.8V - is false. The voltage output of a Li-ion poly battery drops over time, so the calculation shown is only a lower bound approximation of each battery’s maximum capacity.

I don’t have information about the exact chemical composition of these batteries, nor the voltage charts, nor do I know what the upper and lower voltage limits are on the OnePlus 5T charging circuit. However, if we estimate that the voltage drops from 3.8V to 3.0V in a linear fashion (V = 3.8 - .8t, 0 <= t <= 1), we can use integration to arrive at approximately 3600 mAh maximum capacity for the OEM battery and 2900 mAh maximum capacity for the aftermarket battery.

Even without exact numbers, these calculations demonstrate that something is fishy about the Da Da Xiong battery’s mAh claims.

Real-world usage

Anyways, I didn’t buy this shady AliExpress battery just so that I could do a bunch of math. I purchased it to restore my phone’s ability to last all day, and it has definitely succeeded at that. From a qualitative perspective, I now have enough juice to keep my phone’s battery fueled all day until I can recharge it at night.

From a quantitative perspective, AccuBattery reports that the aftermarket battery has an estimated 3360 mAh capacity, which about matches the capacity of the OEM battery:

However, what AccuBattery fails to account for is the fact that once the aftermarket battery reaches 15%, the battery percentage begins to free-fall until it reaches 0% and shuts off. It seems like 15% on the aftermarket battery is equivalent to 1% on the OEM battery. I think this is because the Android OS cannot correctly estimate the battery’s remaining charge because it has different voltage characteristics than the OEM battery, but it doesn’t really bother me, I just have to make sure that to charge the phone at 15% instead of 1%. This seems to be an extremely common experience with DIY battery replacements - even folks using the iFixit battery run in to this issue.

If we take 15% off of AccuBattery’s estimated capacity, we get 2856 mAh, which is really really close to what a brand new OnePlus 5T reports - AccuBattery estimates the OEM battery as having ~3000 mAh capacity when it is brand new. That about matches my experience - with the Da Da Xiong battery, the phone is staying alive longer, almost like when it was new.

Conclusions

Random Chinese batteries do not work as advertised - they will not magically double your phone’s battery capacity.
However, random Chinese batteries work almost as well as brand new OEM batteries, but your battery percentage will forever be miscalibrated.
Never trust OnePlus customer service.

Pilo: Raspberry Pi-Powered Lights-Out Remote Server Management for $60 or less

Thu, 20 Aug 2020 13:46:50 +0000

The completed Pilo controller, before final installation. The USB capture card and Arduino Nano USB serial are soldered to the underside of the 3B+.

Like many geeks, I have a “home server” made from off-the-shelf, consumer-grade PC parts, from which I run my weekend programming projects, game servers for friends, this website, and so on. Recently, I had a power event at the house that caused the server to reboot. When the power came back on, the server booted, but it was stuck at the boot screen waiting for me to enter the disk decryption passphrase!

Luckily, I was at home and asleep at the time. Once I woke up and realized something was amiss, I was able to plug in a keyboard and enter the passphrase. But this event got me thinking - what if I wasn’t home at the time? What if I was in another country? What if someday, I move this server outside of my house and need to regularly access the physical screen and keyboard?

In the “real server” world, the solution to this is known as “lights-out management” (LOM). Every major server manufacturer has their own flavor of this, such as HP’s iLO (Integrated Lights-Out). There are even industry standards like IPMI that define common interfaces for LOM implementations. Commonly supported functions for LOM systems include:

Controlling keyboard and mouse input
Controlling power button status (so you can restart, shutdown, force off…)
Seeing the raw video output from the motherboard (even pre-boot - even for BIOS)
Mounting ISOs as disks

I decided to make my own Raspberry Pi-based LOM that can do some of these things, to help decrease my stress next time I leave my house for an extended period of time. I’d like to introduce Pilo - “Pi Lights-Out”.

Note: This post describes how I arrived at the final design of this system. If you just want the instructions for setting this up on your own, start reading at “Tutorial”.

Building the Keyboard Controller

When I started looking for ways to use my Pi to send keyboard commands to a computer, the problem I discovered was that all of the existing methods rely on using the Raspberry Pi Zero as a USB host, which disables using the onboard USB port for other purposes. Additionally, this method does not work on other boards, like the Raspberry Pi 3B+. This was problematic, because I wanted to use the 3B+ due to the on-board Ethernet - that, and the fact that I had one kicking around from a previous Pi-hole deployment.

A more suitable solution would be to emulate a keyboard via the GPIO pins of the Pi. This would theoretically not affect existing USB devices, and could be used on any model of Pi, not just the Pi Zero. So I started looked into trying to “bit-bang” the USB Human Interface Device protocol via the Pi’s GPIO pins.

I pretty quickly hit a dead end with that. There are many, many existing discussions on the web about bit-banging the USB protocol on the Pi. The consensus seems to be that the Raspberry Pi’s GPIO is too slow to emulate USB, and even if it could output bits fast enough to act as a USB device, the implementation would be extremely buggy because of the non-real-time nature of the Linux OS (read more).

But USB isn’t the only way to send keypresses to a computer - almost a decade before the USB standard was a twinkle in Compaq’s eye, IBM was using the PS/2 standard (not to be confused with the PS2) to connect mice and keyboards to PCs. It’s still not feasible to use the Pi to bit-bang the PS/2 protocol, but we can use an Arduino as a daughterboard, and the ps2dev library can handle the nitty-gritty of the serial protocol for PS/2.

So that’s what I did. The Arduino Nano pictured on the left is plugged directly into the PS/2 combo port on the back of the motherboard (ignore the unused 5V wire, red). Originally, I planned on cutting the end off of a PS/2 cable and making it all nice, but the Goodwill near me didn’t have any PS/2 junk and it turns out that breadboard wires just fit oh-so-snugly into the DIN holes. So this is how it’s gonna be.

The Arduino Nano is flashed with a short program that makes it act as a dumb pipe which blindly shuttles bytes from the Arduino’s USB serial port to the PS/2 connection on the motherboard. This means that all of the logic for which keyboard commands should be sent has to be written on the Pi-side, which is nice, because it means that we should never have to re-flash the Arduino to update some keyboard logic.

Note that currently, Pilo is only built to control keyboard input, since it is oriented towards server use. It would be possible to add PS/2 mouse output with no additional hardware, just 2 or 3 wires for PS/2 mouse CLK, DATA, and GND (unless using combo port). The ps2dev library contains functions for mouse control as well.

Power Control via PS/2

Originally, I thought I was going to have to wire a relay to the motherboard’s RESET pin to allow Pilo to control the computer’s power. This is the approach that diy-ipmi, another similar project, uses. However, while researching the keyboard controller, I rediscovered a long-lost secret of the PS/2 standard: the “ACPI keys”. ACPI, or the Advanced Configuration and Power Interface, is a set of power management standards for PCs. The PS/2 standards define Power, WakeUp, and Sleep key scancodes that can be used to control the power status of the PC - just like the power button on the front of the box, a short-press of the Power key requests the OS to shutdown. However, on my motherboard, a long-press of Power does NOT seem to force the power off.

Almost all BIOS support using the ACPI keys to power the system on from a powered down state. Above is what the option looks like in my BIOS (in ACPI, the powered-off state is known as “S5”). With this option enabled, sending the Power scancode will boot the computer up from an off state.

So, now the Arduino has two responsibilities in Pilo: to send regular keypresses to the computer, and to send power commands to the computer. This saves us from having to install a relay for the motherboard’s RESET pin.

Capturing Video Output

My server has a GPU with HDMI output, so I decided to use a USB HDMI capture card to get the video feed for Pilo. I found one on Amazon for about $15. When connected to the Pi, it acts as a regular USB webcam, available under /dev/videoX.

Before this project, I had no experience with streaming video over the web. I decided to use uv4l (“Userspace Video4Linux”)’s uv4l-server component to set up an HTTP video server. uv4l makes it easy to set up a simple MJPEG stream, which is the goofiest possible video stream - each frame of the stream is a full JPEG image, sent to you in real time. As you can imagine, it’s not the lightest on bandwidth, but it is easy to embed - all web browsers support embedding it in an <img> tag: <img src="/stream.mjpeg"/>

I configured the uv4l-server to only listen on localhost, with the idea that I could reverse-proxy connections to the video stream to provide security.

Creating the application

For the Pilo interface, I decided to go with a web app, instead of something like VNC, or the actual IPMI protocol. This was mostly due to my background in web development, and the fact that the app can be accessed with a web browser, something every computer has installed. Here is a short video showing the completed Pilo app in action:

Your browser doesn't support WEBMs. Download the video instead.

You can find the GitHub repo for Pilo here. It consists of two major components:

frontend - uses vanilla HTML/CSS/JS to display the interface, translate keypresses, and communicate with the server via websockets
server - the HTTP server, written in Node.js. Authenticates requests using HTTP basic auth, communicates with the keyboard controller via the serialport library, and manages reverse-proxying of the uv4l-server video stream

There are also end-to-end tests in the e2e folder which use Cypress to test the application in real web browsers. This runs against Firefox and Chrome on every commit to CI via a GitHub Actions workflow.

The README contains information on building and testing the project if you are interested in contributing. Built packages are also published to npm for production use.

Packaging the Pilo

One of my goals when building Pilo was to make it small enough to fit inside of my server case. Check out these photos to see how it fit in:

Left: Server case before embedding the Pilo. Right: Server case after embedding the Pilo. HDMI, Ethernet, and micro-USB power are routed through the left-most PCI-E slot, while PS/2 is routed through the I/O shield in the top-right. The Pilo itself sits atop a ledge in the bottom-right of the image.

Here are some pictures of how the final assembly was made:

Outer photo: Pi with the USB devices soldered on to the underside to save space. Inset photo: Final package taped up and ready for install, with PS/2 cable coming out. Hot glue and tape make me the solderer I ain’t.

The completed package fits within a bounding box only slightly larger than the Pi itself, about 88mm x 60mm x 20mm, which means the Pilo can fit conveniently into a standard 3.5” hard drive bay.

Tutorial

Parts List

Raspberry Pi 3B+ ($25 at MicroCenter, $35 everywhere else)
- Or other micro linux computer - even a Pi Zero could work, but you’d be using WiFi, and you’d need a USB hub for the Arduino serial, or set up the Arduino serial via GPIO
Arduino Nano (Amazon has them at $16.99 for 5 - $3.40 each)
- Any other 5V-logic-level Arduino would work as well. Needs to be 5V or have the CLK + DATA outputs converted from 3V3 to 5V, since the PS/2 serial connection expects 5V.
USB HDMI Capture Card (video4linux compatible - most cards are) (Amazon, $13.99)
(optional) PS/2 plug, to make a tidy connection
Supplies: Wires, soldering iron if you need to solder, microSD card and micro-USB power supply for the Pi…

Comes out to roughly $60 if you buy everything at it’s cheapest, $50 if you’re lucky enough to live near a MicroCenter.

Setup

Before getting started, make sure you have your devices ready:
- Have a Debian-based OS like Raspbian or DietPi installed on the Pi’s microSD card.
- Flash the serial_to_ps2 program to your Arduino Nano.

Connect everything according to this beautiful wiring diagram:

 ┌───────────────────────────┐
 │                   Ethernet├───────────────────────────┤Ext. Ethernet
 │ RasPi 3B+     USB Power In├───────────────────────────┤Ext. 5V Power
 │ (or other)        USB Port├───────────┐
 │                   USB Port├──────┐    │
 └───────────────────────────┘      │    │
                                    │    │ ┌────────────────────────────┐
 ┌───────────────────────────┐      │    └─┤USB Plug  HDMI Capture Card │
 │ Server            HDMI Out├──────┼──────┤HDMI In                     │
 │         PS/2 Keyboard Port├───┐  │      └────────────────────────────┘
 └───────────────────────────┘   │  │
                                 │  │
                                 │  │      ┌────────────────────────────┐
                                 │  └──────┤USB Port                    │
                                 │      ┌──┤D2 (DATA)      Arduino Nano │
                                 └──────┤──┤D3 (CLK)        (or other)  │
                                        └──┤GND (GND)                   │
                                           └────────────────────────────┘

Connect D2 (DATA), D3 (CLK), and GND from the Arduino Nano directly to the respective pins of the PS/2 keyboard port on the motherboard. Consult the Internet for a reference pinout.
- You can dress this up by scavenging a spare PS/2 pigtail and stripping the end, but breadboard wires also fit snugly into the DIN port.
Turn off your server before connecting or disconnecting anything to/from the PS/2 port. The PS/2 port was not designed with hotplugging in mind. You risk damaging your components and your OS most likely won’t detect a hot-plugged PS/2 device, so don’t bother.
Do not connect the +5V pin from the PS/2 port to VIN on the Arduino - this is a bad idea for a myriad of reasons.

Set up the MJPEG streaming server for the USB HDMI capture card.
1. Follow the instructions on the uv4l website to install the apt sources for uv4l. Note for Debian Buster users: There is not a uv4l apt repo for Buster yet, but the Stretch repo seem to work fine on Buster.
2. Update the apt index, and install the required packages:
```
 apt update
 apt install uv4l uv4l-server uv4l-uvc uv4l-mjpegstream
```
3. Retrieve your USB HDMI capture card’s ID by running lsusb. It should be a hexadecimal string like 1a2b:3c4d.
4. Start the uv4l server listening on 127.0.0.1:9000, substituting your device ID for 1a2b:3c4d below:
```
 uv4l --driver uvc --device-id '1a2b:3c4d' --auto-video_nr --server-option '--port=9000' --server-option '--bind-host-address=127.0.0.1'
```
  You should be able to access a MJPEG stream of the USB HDMI capture card locally on the Pi at http://127.0.0.1:9000/stream/video.mjpeg. You will not be able to access this over the network, since it is bound to localhost by the --bind-host-address.
5. Set the command from (4) to run on every boot, for example, by adding it to /etc/rc.local.
Now, it is time to generate your auth credentials. Follow this algorithm:
1. Think of a secure username and a password. For example: foo, bar
2. Join them with a colon. For example: foo:bar
3. SHA256 the result. For example: a765a8beaa9d561d4c5cbed29d8f4e30870297fdfa9cb7d6e9848a95fec9f937
4. This is your AUTH_SHA.
Install Node.js and npm:
```
 apt install nodejs npm
```
Install the latest distribution of pilo from npm:
```
 npm install --global pilo
```
You may need to pass --unsafe-perms if installing as root.
You can now start pilo on port 3000 by running the command with AUTH_SHA set in an environment variable:
```
 AUTH_SHA=a765a8beaa9d561d4c5cbed29d8f4e30870297fdfa9cb7d6e9848a95fec9f937 pilo &
```
You should now be able to access Pilo at http://<your-pi-IP>:3000/, using the username and password you created earlier.
Add the above command to /etc/rc.local to run it on boot.

Creating a Site-to-Site WireGuard VPN for a home server

Sat, 02 Nov 2019 13:17:40 +0000

This guy is looking at pictures of my wife, probably. photo source

For the last decade or so, I’ve been steadily increasing the amount of data I send to the cloud. I sync photos of my friends and family to Amazon Photos, blast my private data off to Microsoft OneDrive, give my passwords to 1password, and trust my web hosting provider not to run away with my data.

I’ve been growing less satisfied with the privacy options afforded by major cloud providers. Amazon Photos, for example, has started using machine learning to identify the people in my photos. I’m not really keen on the mental image of Jeff Bezos, sitting on a yacht, looking at pictures of my wife.

For that reason, I’ve decided to start moving my personal data into my personal control. I want the bits and bytes that describe the intimate details of my life to live around under my own roof and only escape to the Internet with my permission.

I’d like to self-host NextCloud (to replace Amazon Photos and OneDrive), static site hosting (to replace my existing VPS and GitHub Pages), and continuous integration using GitLab CE (to replace Travis CI). It’s also a pipe dream to one day host my own email server, so I can move off of Google Apps.

To make all that possible, I’d need a way for Internet traffic to reach my home LAN, behind a router with a dynamic IP. I don’t want to use dynamic DNS, since I think it provides a poor user experience due to DNS caching. Also, my Internet service provider does not explicitly allow server hosting, so excess incoming Internet traffic might get me in trouble. 😅

The solution comes in the form of an Internet-facing server with a static IP. That server will receive requests and forward them to the LAN server through an encrypted, performant WireGuard tunnel:

┌-------------------┐     ┌------------------------┐
| Internet Traffic  | <-> | Internet-Facing Server |
└-------------------┘     └------------------------┘
                                    /\
                                    || WireGuard Site-to-Site VPN
                                    \/
                             ┌-----------------┐
                             | Home LAN Server |
                             └-----------------┘

I chose WireGuard over other VPN candidates because of the simplicity of configuration and low server overhead. Without further ado, let’s get into how to set this up.

Step 1: Internet-Facing Server Setup

When choosing a server provider for your Internet-facing server, make sure to choose one with low latency to your home network, since that latency will be added to every request you make.

If the provider has test servers listed on their website, you can ping them from your home network to make an estimate of the round-trip-time that will be added to each request.

I chose RamNode for my hosting, since I get about 3ms of ping to their test IP in Atlanta:

~ ping -c 4 107.191.101.180
PING 107.191.101.180 (107.191.101.180) 56(84) bytes of data.
64 bytes from 107.191.101.180: icmp_seq=1 ttl=55 time=3.08 ms
64 bytes from 107.191.101.180: icmp_seq=2 ttl=55 time=3.34 ms
64 bytes from 107.191.101.180: icmp_seq=3 ttl=55 time=3.08 ms
64 bytes from 107.191.101.180: icmp_seq=4 ttl=55 time=3.41 ms

--- 107.191.101.180 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 7ms
rtt min/avg/max/mdev = 3.076/3.226/3.410/0.165 ms

Since WireGuard is really efficient, you don’t need a beefy, expensive server to run it on. I chose a server with 512MB of RAM, 1 CPU core, and 2 TB of outgoing bandwidth per month for $3/mo. This will be the only real expense of this project.

I installed CentOS on my Internet-facing server, but WireGuard is compatible with a wide variety of operating systems.

Once you have your server, SSH in and follow this guide to configuring WireGuard:

Install WireGuard by following the instructions for your server OS.
After installing WireGuard, you will have access to the wg command, which we will use to generate public/private keypairs for the server and client.
- Run wg genkey to generate a private key. This will be the server’s private key. This should be kept a secret, as it can be used to decrypt data sent to the server.
- Now, pipe that result into wg pubkey to generate the server’s public key. This will used later to configure the client to send encrypted messages to the server. For example: echo "server-private-key" | wg pubkey
- Repeat the above steps to generate a private & public key for the LAN client.
Create a file using your favorite text editor in /etc/wireguard/wg0.conf, and fill it out using the below template. If you’re curious about the wg0.conf file format, check out the wg-quick man page for more information.

Now that you’ve configured the server, you can bring up the WireGuard interface by doing wg-quick up wg0.

Do wg show to see the status of your WireGuard network:

~ wg show
interface: wg0
public key: your-server-public-key
private key: (hidden)
listening port: 51820

peer: your-client-public-key
allowed ips: 10.222.0.2/32
persistent keepalive: every 25 seconds

Now use systemctl enable wg-quick@wg0 to ensure that this interface is brought up on every boot.

Congrats! Your Internet-facing server is now set up to act as a WireGuard host. Now let’s proceed to the client configuration on the LAN server.

Step 2: LAN Server Setup

Follow these instructions on your home LAN server to set it up as a WireGuard client:

Install WireGuard using the installation instructions for your OS.
Create a file using your favorite text editor in /etc/wireguard/wg0.conf, and fill it out using the below template. Again, for more info on the wg0.conf file format, check out the wg-quick man page.

Now that you’ve configured the client, you can bring up the WireGuard interface by doing wg-quick up wg0.

Do wg show to see the status of your WireGuard network:

~ wg show
interface: wg0
  public key: your-client-private-key
  private key: (hidden)
  listening port: 55018

peer: your-server-public-key
  endpoint: your-server-domain-name-or-IP-address:51820
  allowed ips: 10.222.0.0/16
  latest handshake: 1 minute, 41 seconds ago
  transfer: 959.23 MiB received, 1.57 GiB sent
  persistent keepalive: every 25 seconds

At this point, you should be able to do ping 10.222.0.1 to reach your WireGuard server through your new VPN.
Now use systemctl enable wg-quick@wg0 to ensure that this interface is brought up on every boot.

Now your VPN is set up and you are ready to start exposing services on your home server through your VPN.

Step 3: Start Exposing Services

You’ll need a way to proxy traffic that hits your Internet-facing server through the VPN to your home server.

For HTTP traffic, set up a reverse proxy on the Internet-facing server. My tool of choice for this is nginx, which has a fantastic reverse proxy module. Here’s a very basic nginx config to proxy traffic for example.com to port 8080 on your LAN server:
```
server {
  server_name example.com;
  location / {
    proxy_pass http://10.222.0.2:8080/;
  }
}
```
For other TCP/IP traffic, set up rinetd on the Internet-facing server. It will tunnel TCP traffic on one port/interface to another port/interface. For example, if you have an IRC server running on port 6667 of your home server, you could put this in /etc/rinetd.conf to forward traffic from port 6667 of the Internet-facing server:
```
# bind to all interfaces on 6667 and pass to LAN server
0.0.0.0 6667 10.222.0.2 6667
```

With both of these methods, keep in mind that the IP of the original client will be obscured by the reverse proxy. You’ll need to use other methods (such as an X-Proxied-For header containing the real client’s IP address) if you want to receive the client’s real IP at your home server.

Now you can start moving all of the services you want to self-host under your own roof! In future articles, I will discuss setting up your own self-hosted photo storage, continuous integration pipelines, web hosting, and others.

Extra: Securing Your Internet-Facing Server

One of the benefits to this setup is that you no longer need to expose your Internet-facing server’s SSH port publicly. You can use the VPN to access it instead.

Set up your computer as a WireGuard client using the same method that you used to set up your home LAN server as a client. Or, just use your home LAN server as a bastion host, so you must be SSH’d into it to SSH into your Internet-facing server.

Set up ufw on your Internet-facing server using these commands:

  # turn on ufw
  ufw enable
  # allow inbound access to WireGuard's port
  ufw allow 51820/udp
  # allow VPN IPs to access SSH on port 22
  ufw allow from 10.222.0.0/24 to any port 22 proto tcp
  # remove default SSH allow rules
  ufw delete allow SSH
  ufw delete allow 22/tcp

Now you should only be able to access SSH on your Internet-facing server via the VPN IP address, 10.222.0.1.

Extra: Alternative WireGuard Distributions

The official WireGuard distribution comes as a kernel mod. While the official implementation is best, there are also some alternatives that run in userspace, if you’re unwilling/unable to install a kernel mod:

wireguard-go - This is WireGuard’s official userspace implementation, written in Go. Recommended.
wireguard-rs - Another userspace implementation, also by the WireGuard authors, written in Rust. WIP, not recommended for production.
boringtun - Cloudflare’s unofficial userspace WireGuard client, also written in Rust. Used in their proprietary Argo Tunnel Site-to-Site VPN. Note: the original author of WireGuard, Jason A. Donenfield, has expressed some opinions about Cloudflare’s involvement in WireGuard.

Dockerizing a Python 3 Flask App Line-by-Line

Fri, 20 Jul 2018 00:00:00 +0000

If you’re like me, you end up writing a lot of Flask apps for random web applications and APIs. When you’re done building these Flask apps, typically, you need to deploy them to production somehow. Unless you’re using a container service like Heroku, deployment is synonymous with SSHing to a live server, installing your dependencies, and starting your application from source.

Docker makes the process of deployment easier by containerizing your app and automatically installing dependencies from a simple declarative config called the Dockerfile. In your Dockerfile, you can specify the operating system to run on, the apt-get or yum commands to run before your application starts, or any other logic you can think of. Docker will generate an image from this and handle running your application using that image inside a container.

In this guide, we’ll walk through the process of writing a Dockerfile for a modern Flask app. By the time we’re done, we’ll have a nice Ubuntu + nginx + uwsgi + Flask stack all working.

Requirements

Before we begin, ensure that you have Docker Community Edition installed. Instructions vary by operating system but it is available for Windows, OS X, and Linux.

Also, you’ll need a Python 3 Flask application to deploy. Make sure that in your app.py , any app.run() calls are wrapped in an if __name__ == '__main__' check so that uwsgi does not accidentally spawn 2 copies of your server.

The Dockerfile

First, create a file named Dockerfile (no extension) in the root of your Flask application and open it in your favorite text editor.

Docker uses a very simple, declarative language to define the build process. Let’s start off by identifying the operating system we’d like to use:

FROM ubuntu:18.10

This tells Docker to fetch the Ubuntu 18.10 Cosmic Cuttlefish disk image from the Docker official repository and use it as the base OS for this container. Next, let’s add some information about the maintainer of this package:

LABEL maintainer="Zach Bloomquist <zach@bloomqu.ist>"

Docker doesn’t use this information for anything except for setting the author field of the created image. It’s also nice to leave a breadcrumb for developers who may come after you. Next:

RUN apt-get update
RUN apt-get install -y python3 python3-dev python3-pip nginx
RUN pip3 install uwsgi

RUN commands are executed while building the Docker image. These will update apt’s package index and then fetch our dependencies. The package python3-dev may stand out to you — this package is required for uwsgi to build when we install it with pip .

COPY ./ ./app
WORKDIR ./app

The COPY command copies files from the source’s filesystem to the container’s filesystem. These commands copy over the application’s source code to a new folder and cd into it for the rest of the build. Next:

RUN pip3 install -r requirements.txt

This installs the requirements for your Python 3 app to execute, assuming you list your dependencies in requirements.txt.

COPY ./nginx.conf /etc/nginx/sites-enabled/default

This command sets up the configuration for nginx inside the container by overwriting Ubuntu’s default. It assumes you have a proper nginx configuration at ./nginx.conf containing something like this:

server {
  location @flask {
    include uwsgi_params;
    uwsgi_pass unix://tmp/uwsgi.sock;
  }
  location / {
    try_files @flask;
  }
}

This example config will pass all requests to the container’s port 80 to the uwsgi application listening on that socket.

Now, back to the Dockerfile:

CMD service nginx start && uwsgi -s /tmp/uwsgi.sock --chmod-socket=666 --manage-script-name --mount /=app:app

The CMD command tells Docker what command to execute when someone runs the image our Dockerfile creates. In this case, we want to start nginx, then start up uwsgi to back it.

Building & Running

Now that our Dockerfile is created, let’s build the image from the current directory:

docker build -t my-image-name .

You can watch Docker chew through installing Ubuntu, installing the system packages, and installing our pip requirements. Once that’s done, you’re ready to boot up a new container based off your image:

docker run -d -p 1337:80 my-image-name

The -d option tells Docker to run the container in the background and print out the container ID. -p 1337:80 maps port 1337 on the host machine to port 80 in the container.

Now that your container is running and port 1337 is mapped, you should be able to visit [http://localhost:1337/](http://localhost:1337/) and see your Flask application running. Woohoo!

You can use the container ID printed out by docker run -d to manage the life cycle of your app. Here are some good CLI commands to know:

docker image ls — list available images
docker container ls — list all containers
docker logs <partial container ID> — tail logs from a container
docker kill <partial container ID> — kill execution of a container
docker restart <partial container ID> — restart container
docker start <partial container ID> — start stopped container
docker stop <partial container ID> — gracefully end container
docker container prune — delete all non-running containers

I hope this guide was helpful to you in Dockerizing your Flask application. If you have any comments or questions, please leave them below.

Appendix: Next Steps

If you really want to turbo-charge your Dockerization, try these tips:

Make use of named containers so you don’t have to keep track of container IDs. Pass the --name my-awesome-name to docker run to name your new containers, then you can do things like docker restart my-awesome-name.
Create shell script helpers to help your users interact with the Dockerized app. For example, you might create a shell script called install.sh that ensures Docker is installed, then builds the Docker image and ensures it runs on startup.
Use Docker Compose to further abstract your deployment. Docker Compose allows you to orchestrate multiple containers for one deployment. In this example, we might’ve used a container just running nginx to proxy requests to a separate container running our Flask app.

Moving From React to Preact: A Developer’s Story

Tue, 17 Jul 2018 00:00:00 +0000

For the past few months, I’ve been working on a single-page application built using Facebook’s React framework.

Those who know me know that I’m obsessed with performance. Getting me on the React bandwagon took years simply because I didn’t like the bloat it introduces. Even though I’ve grown to love React for the deftness it affords when designing front-ends, I still have an issue with all the bloat it introduces.

Recently, I heard about the Preact project — a drop-in, API-compatible replacement for the React framework that’s only 3kb in size compared to React’s 135kb. It’s faster and easier to understand than React. When I heard about this framework, I knew I had to try it. I’ve documented my efforts here (including some troubleshooting tips) to guide others who would follow this path.

The React App

Before transitioning to Preact, I wanted to examine my existing app so I could be aware of any potential pain points I’d encounter while transitioning.

My application was created with create-react-app. This means a lot of the complexity of the build environment is hidden away in packages like react-scripts. This is good for a new user, but potentially bad when trying to switch away from React.

I make heavy use of React’s createContext API to enable authentication and back-end use throughout the app. Preact only supports the legacy context API, so I needed to use Georgios Valotasios’s preact-context library to provide those APIs.

My app uses react-router for routing and passing state between linked components. I had some concerns about this working.

s/React/Preact/g

Preact has a helpful guide for switching. They provide a shim, preact-compat, that allows you to swap your codebase over with minimal code changes. There’s also a longer guide to migrating your codebase to use preact without the compat libraries.

I chose to do the full migration in order to take advantage of the full performance benefits — the preact-compat library adds 2kb to Preact’s 3kb size. I wouldn’t have blinked at 2kb before, but now the bloat of React is no longer in the way, 2kb seems like a whole lot more.

First, install the preact package and get rid of react:

yarn add preact
yarn remove react react-dom

This will make all of your React plugins begin to throw warnings about unmet peer dependencies, this is fine. Currently there is no way to tell these plugins about Preact.

Then, use the following command to replace all the import X from 'react' statements in my package with import X from 'preact' (assumes all your code is in src):

find ./src/ -type f -print0 | xargs -0 sed -Ei "s/(['\"])react(-dom)?(['\"])/'preact'/g"

The import in index.js must be corrected manually. Replace the React and ReactDOM imports with

import React, { render } from 'preact';

Preact moves the render method out of ReactDOM and into its own export, so change the ReactDOM.render call to just a call to render .

To fix React Contexts, install preact-context with yarn add preact-context. Then, add the import statement to all your Contexts:

import { createContext } from 'preact-context';

After changing React.createContext calls to createContext calls, all that’s left to do is run yarn start and pray.

You’re done!

(In a perfect world,) your app should now be working with Preact instead of React. Kudos to the developers of Preact for creating a drop-in replacement for the temperamental React framework. Enjoy your faster load times, easier-to-debug code base, and lower bandwidth usage!

If you did encounter issues, continue on to Troubleshooting, where I’ve outlined a few of the issues I ran into when doing this on my app.

Troubleshooting

Of course, my migration did not go flawlessly — things never seem to, do they? I hope yours does, but if it does not, here are the issues I encountered and how I overcame them.

Context changes

My first issue was that changes being broadcast from a Context were no longer being sent to child components. I had the Context set up like this:

<Provider value={this.state}> ... </Provider>

Context Consumers are supposed to re-render whenever the Provider’s value changes. In React, the above line would cause all dependent components to re-render when the Provider’s state changed.

In Preact, this stopped working. I’m assuming this is because Preact does not copy state on setState; instead, it modifies state in place. Because JS object comparisons are done by reference, no change appears to happen and so the Consumer components are never re-rendered.

To get around this, I’m copying state to a new object whenever it changes:

<Provider value={Object.assign({}, this.state)}> ... </Provider>

If anyone knows a cleaner way, please let me know.

React Developer Tools

I discovered that my React developer tools were no longer working. This is because Preact does not include the developer tools in the main bundle to cut down on unnecessary load time.

To fix this, I just added the following line to index.js to load the debug module if this module is being hot-reloaded by Webpack:

if(module.hot) require('preact/debug')

React Router errors

The last issue I encountered was a problem with react-router-dom. The BrowserRouter component was throwing the following error during render:

Warning: Failed prop type: Invalid prop `children` supplied to `Router`, expected a ReactNode.

For some reason, the <div> element that was the immediate child of my BrowserRouter was not being recognized as a valid component. I tracked down this error to the definition of isValidElement in prop-types:

var REACT_ELEMENT_TYPE = (typeof Symbol === 'function' &&
  Symbol.for &&
  Symbol.for('react.element')) ||
  0xeac7;

var isValidElement = function(object) {
  return typeof object === 'object' &&
  object !== null &&
  object.$typeof === REACT_ELEMENT_TYPE;
};

The <div> fails the isValidElement test because it has no $$typeof property. The only thing I can seem to figure out about $$typeof is that it’s a special property React uses in order to distinguish React-created DOM objects from native DOM objects.

I tried adding a hook to preact to automatically populate the $$typeof on new VNode (virtual nodes) in the DOM, but that just uncovered another host of issues related to differences between the React virtual DOM and Preact’s VDOM.

So, with a heavy heart (it’s 2 kilobytes of code!!), I decided to install preact-compat (2kb!!!) and patch it into my app.

Doing this was fairly simple, just yarn add preact-compat and add 2 dependencies to your resolve.alias configuration in your webpack config:

alias: {
  'react': 'react-compat',
  'react-dom': 'react-compat'
},

Note: Because I created my app with create-react-app, I had to run yarn eject so that I could edit my webpack config instead of using one from node_modules. You may or may not have to do this first.

After doing this, react-router-dom worked like a charm.

I also went through my project and replaced all my preact imports with preact-compat just to match the documented way of doing it:

find ./src/ -type f -print0 | xargs -0 sed -Ei "s/'preact'/'preact-compat'/g"

A survey of crossdomain.xml vulnerabilities

Fri, 15 Aug 2014 07:15:00 +0000

Vulnerable crossdomain.xml files can be used by malicious people to run CSRF attacks if the victim has Flash installed on their computer. In response to a post by chs on crossdomain.xml proofs of concept and Seth Art’s real-world exploit of Bing using crossdomain.xml, I created an application in Ruby which parses the Alexa top million site list (CSV, 10MB) and scans for vulnerable crossdomain.xml files. Vulnerable here is defined as a crossdomain.xml file which permits connections from any domain name (*). It sorts the domains into four categories:

Unable to connect: Ruby was unable to establish a connection to the website. Interestingly enough, a significant portion of Alexa’s top million sites were inaccessible during this survey.
Invalid or 404: Returned 404 or the returned XML was not valid.
Secure: The XML returned does not contain a reference to allow-access-from domain=”*”. This does not necessarily mean that the whole crossdomain.xml file is secure, just that it is not vulnerable to the most basic of CSRF exploits.
Permissive: The XML returned from a GET to /crossdomain.xml does allow access from any domain.

Without further ado, let’s get into it.

The Code

I chose Ruby for this project because it has good XML processing libraries, is reasonably fast, and because I needed an excuse to practice Ruby.

require 'net/http'
require 'rexml/document'
include REXML
require 'csv'

counters = {
	'unconnect'   => 0,
	'invalid-404' => 0,
	'permissive'  => 0,
	'secure'      => 0,
	'total-count' => 0
}

trap 'SIGINT' do
	print counters.inspect
	exit 130
end

permissive = CSV.open('permissive.csv','wb')

CSV.foreach('top-1m.csv') do |row|
	counters['total-count'] += 1
	print "\n"+'Getting '+row[1]+'... '
	begin
		xd = Net::HTTP.get(row[1], '/crossdomain.xml')
	rescue
		counters['unconnect'] += 1
		print 'unable to connect'
		next
	end
	begin
		xd = REXML::Document.new(xd)
	rescue
		counters['invalid-404'] += 1
		print 'invalid xml'
		next
	end
	wildcard_access = false
	XPath.each(xd,'//allow-access-from') do |access|
		next unless access.attributes['domain'] == '*' # <allow-access-from domain="*">

		wildcard_access = true
		counters['permissive'] += 1
		print 'permissive'
		permissive << row
		break
	end
	unless wildcard_access
		counters['secure'] += 1
		print 'secure'
	end
end

print counters.inspect

The Results

After 160,169 websites were inspected over the course of a few days, the script hung.

3,535 (2.2%) of the websites were down at the time of the scan.
84,883 (53%) of the websites had invalid or non-existent XML files at /crossdomain.xml.
67,097 (41.9%) of the websites surveyed had a “secure” crossdomain.xml file.
4,653 (2.9%) of the websites surveyed had insecure crossdomain.xml files.

A wildcard crossdomain.xml file is fine for certain websites, but a quick scan of the results reveals a number of banks, bitcoin websites, and popular entertainment sites (9gag and Vimeo included) with poor crossdomain.xml files. The results as a CSV with columns corresponding to the Alexa rank and the domain name.

Although a full scan of the Alexa top million was not completed, an alarmingly large number of sites have overly permissive and insecure crossdomain.xml files.

spoofident: A fake identd written in Python

Sat, 21 Jun 2014 20:27:25 +0000

Many protocols such as IRC require or strongly suggest the use of an ident daemon to prove that you are who you say you are, or to hold you accountable for your actions. An identd is supposed to respond to queries as to which user is using which port; however, this information can be potentially dangerous. A real identd allows attackers to gain information about your system - usernames, active ports, even a fingerprint of your active operating system. The RFC linked above even cites these vulnerabilities.

I had a need to run an ident server; however, I am wary of creating unnecessary security holes in my server. That’s why I wrote spoofident. spoofident is a daemon written in Python which provides a custom username/OS response to all incoming ident queries. It is dual-stack (meaning that it runs on both IPv4 and IPv6) and written to consume little resources, less than oidentd. I suggest using it if you are in a situation where you need to provide ident but refuse to compromise the security of your systems for that functionality.

GitHub repo for spoofident

README for spoofident

Defeating Comcast BitTorrent Throttling: The Easy Way

Sat, 21 Jun 2014 00:07:10 +0000

If you torrent a lot, eventually Comcast/xfinity will throttle your torrent speeds to 20kbps or below. Luckily, there is a simple fix which works without installing any external applications. Simply configure the listening port in your BitTorrent client to be 443, and ensure that the port is open on your router. This works because Comcast’s deep-packet inspection ignores packets on common Internet ports to save processing power, and 443 is the HTTPS port. This fix will not disrupt your HTTPS traffic. This also works on port 80 (HTTP) and port 53 (DNS).

Fixing a corrupted TrueCrypt volume for Windows rescue

Sun, 15 Jun 2014 12:19:56 +0000

Like an idiot, I held down the power button to reboot my machine. It booted back up, and once I entered my TrueCrypt full-disk encryption password, I was greeted with a Windows rescue and repair boot screen. I managed to create a system volume which was both encrypted and corrupted, meaning that Windows rescue could not read the volume. Instead, it asked me to insert a driver disk for my hard drive so it could attempt rescue. If I entered the command line and used DISKPART to try to list volumes, it displayed C: as filesystem type RAW.

Not a good thing.

Happily enough, this is an issue with a simple solution. Insert your TrueCrypt rescue disk which you created when you originally encrypted your system and reboot. Press F8 for “Repair Options” when you get to the TrueCrypt authentication screen, and then press 1 to permanently decrypt your system disk. At this point, you’ll need to enter your decryption key, and decryption will begin. On a 500GB laptop drive, this process took over 8 hours to complete (!), so here are some other options you could try before attempting to decrypt the drive:

Create a bootable live Linux thumb drive, install TrueCrypt onto it, and mount your system volume. From there, you can use any of the Windows repair tools available on Linux or do a chkdsk or what have you. TrueCrypt on Linux cannot mount partially decrypted volumes, so if you want to go this route, be sure not to begin decryption through the TrueCrypt rescue disk
Create a bootable WinPE drive and use it to repair Windows. I did not try this one so I do not know if TrueCrypt can handle partially decrypted volumes on WinPE.
Manage to get TrueCrypt running within the Windows rescue environment so the volume is mountable and rescuable. Windows rescue kills it, saying that “This image cannot run because the appropriate subsystem is not loaded.” This happens because Windows rescue is not running a full-fledged copy of Windows, so the subsystem is not available. If you have an afternoon to spare for fun Windows finagling, this is the option for you.