Blaine's World

DNS Web Hosting

Published

An experiment that uses DNS TXT records to act as a rudimentary object store for a small website. Why pay for hosting when you’re already paying for a domain name?

While in the shower I had a thought: is it possible to host a website without anything more than a domain name? Not just a snippet of HTML, but and entire site? How would anyone access such a site? I could stuff data into DNS TXT records, but they must not offer any appreciable amount of storage… right?

Feasibility

After some internet searching, I found that TXT records can hold a list of character-string values. Each string can be up to 255 characters long, but multiple strings can be set to form longer values. I couldn’t find an upper limit published for how many strings could be chained, so I started poking at my DNS provider to see what limits they imposed.

I use Namecheap, who has a nice web interface for setting records. Splitting of long TXT strings into a multi-string value is handled automatically on their backend, so I started pasting longer and longer random string values until I got an error around ~3k characters. I wanted an official answer, so I did some more searching and ran into documentation on limits which states each record can be 2500 characters with up to a total of 150 records per domain. Using nothing but TXT records would allow 375,000 characters to be stored, not too shabby.

Record Format

A decent amount of data can be stored, but how? A full website consists of many different file formats, some binary and some plain text. Files could be stuffed directly into records as-is, but storage of non-printing characters may not be supported by all DNS servers and processing becomes a pain with shell scripting. A safer bet would be to encode files into the printable ASCII range and assign a filename prefix to each record using a common key=value format.

Binary data can easily be converted to printing characters using base64 encoding, but this introduces a 33% overhead that eats away at the already low record length limit. Compressing the file before base64 encoding it can counter this overhead if the input file contains sequences that can be compressed.

I did some searching and landed on the well-tested and commonly available gzip (LZ77) compression algorithm. There are other methods that may offer better size reduction, but gzip has the advantage of being simple and “stream-oriented” (it’s not an overly-complicated container format like xz that has its own overhead issues with small payloads).

So, now I had a record format that could store the data contained within a file and the filename that contains it:

<filename>=<base64 encoded gzipped file content>

This record format has some advantages:

  • Filename is specified – a full directory of files can be stored
  • All data is plain text – wide compatability with DNS server implementations
  • Compression is utilized – some or all base64 overhead can be countered

As well as some drawbacks:

  • Files must fit into a single record – asset size is limited by the DNS host
  • All files are gzip compressed – binary data may end up larger if it’s not very compressible

Those drawbacks could be addressed with a more sophisticated storage scheme, but I chose the simple route to reduce the complexity of the client needed to access the stored data.

The gzip format is also capable of storing the filename, but instead the filename is prepended to the beginning of the TXT record to make it more human-readable and to reduce record size (the FNAME member field isn’t compressed and ends up taking up more space than the raw filename after being base64 encoded).

Site Access

How does someone even access a site stored in a TXT record? No browser will (currently) load arbitrary TXT using Javascript, so bootstrapping with another HTTP-served static site is a no-go. The audience for this experiment is already niche, so a shell script incantation seemed appropriate. Something to download the files stored in records to a local directory and open a browser to view them:

#!/bin/sh

# set the domain to pull TXT records for
SERVER=1.1.1.1
DOMAIN=dnsweb.blaines.world
PAGE=index.html

# this function extracts a single TXT file record to a file
# $0 = filename
# $1 = base64 gzipped data
extract_record() {
    # missing arguments indicate xargs was ran on empty input
    if [ "$#" -ne 2 ]; then
        echo "No website data found :(" >&2
        exit 1
    fi

    # decode base64 to gzip and extract to a file
    printf "$2" | base64 -d | gzip -d -c > "$1"
}
# make it available in subshells (for use by xargs)
export -f extract_record

# create a directory to hold downloaded files
mkdir "$DOMAIN"
cd "$DOMAIN"

# download each key=value record to a file
dig +short "@$SERVER" "$DOMAIN" TXT     \   # fetch TXT records
    | sed -r                            \   # use extended regex
        -e '/^"[a-zA-Z0-9._-]+=H4s/!d'  \   # filter out non-files
        -e 's/^"|" "|"$//g'             \   # join multi-string records
        -e 's/=/\n/'                    \   # split record into 2 lines
    | xargs -L 2                  \   # process each record pair
        sh -c 'extract_record $@' _   # call extract_record with pair

# open the downloaded page with the preferred application
# try xdg-open (linux/unix) then open (macos) as a backup
if [ -f "$PAGE" ]; then
    echo "Opening page with default application..."
    (xdg-open "./$PAGE" || open "./$PAGE") >/dev/null 2>&1
fi

Note that the above script is annotated in a way that makes it an invalid shell script (you can’t just paste and run it as-is). It’s short enough that it can be made into a one-liner that will fit in a TXT record itself (which can be pasted as-is):

_S=1.1.1.1; _D=dnsweb.blaines.world; _P=index.html; _e() { if [ "$#" -ne 2 ]; then echo "No website data found :(" >&2; exit 1; fi; printf "$2" | base64 -d | gzip -d -c > "$1"; }; export -f _e; mkdir "$_D" && cd "$_D" && dig +short "@$_S" "$_D" TXT | sed -r -e '/^"[a-zA-Z0-9._-]+=H4s/!d' -e 's/^"|" "|"$//g' -e 's/=/\n/' | xargs -L 2 sh -c '_e $@' _; [ -f "$_P" ] && echo "Opening page with default application..." && (xdg-open "./$_P" || open "./$_P") >/dev/null 2>&1

The command dig is used to pull TXT records from a specific public DNS server. While initially testing this out with a friend, we noticed that their self-hosted DNS server implementation returned empty results when querying for TXT records. At first, I assumed this was just because the self-hosted server didn’t support TXT records, but after poking at public DNS servers I noticed the same behavior from Quad 9 and Google’s 8.8.8.8. Pulling the same records through Cloudflare’s 1.1.1.1 presents them in a way that dig can understand, so this is used as a workaround.

While troubleshooting this, I directly queried one of the official Namecheap servers (dns1.registrar-servers.com) with dig and got a malformatted message error! I also got an error with nslookup so I monitored the same traffic with wireshark which showed a generic error about an invalid DNS response. I was able to successfully pull TXT records for my other domains using the same official server, but those domains have many fewer TXT records – none of which are large enough to span multiple DNS character-strings.

I suspect that Namecheap is doing something out of DNS spec when handling long TXT records that span multiple character-strings and that some intermediate (like an ISP DNS server or even systemd-resolved) is gracefully handling the data and re-presenting it in a way that standard tools understand. This is a guess, but could explain why only long records are affected and why the data can still be retrieved via the use of an intermediate DNS server.

The magic value prefix H4s indicates the payload is gzip data; this is the value of ID1, ID2, and the upper 2 unused bits of CM in the first member of the data that is outout by the gzip command. Filtering on this value allows other non-file records to co-exist on the same domain (which provides way to include the access script and instructions as additional TXT records).

Demo

I threw together a small demo page that will be downloaded and opened when the shell script in the above section is ran. This page is heavily stylized and showcases some pictures of my cats because I wanted to see what kind of images and assets I could cram into such a small space.

Note the above screenshot is intentionally bad to encourage direct access.

Thoughts

The title of this experiment implies use for website hosting, but this type of storage could be used for any small file payload. An interesting follow-on project could be the creation of a web server that handles unpacking of TXT records on the back-end; this could act as a bridge between the normal web and the web hidden in DNS TXT records.

And, while long TXT records are supported by the DNS protocol, the implementation of my particular service provider leaves much to be desired. It’s a bummer that records need to be queried through a third party in order to be usable with dig and other tools. Software is mutable, though, so this may be corrected if it is truly a bug.

I know I’m not the only person who has abused DNS like this. While writing this up I ran into another experiment that explored a similar execution using zlib compression and a 255 character limit for each record to obviate the need for long multi-string records. I’m sure there are others running around as well.

Links

  • Site Generator Makefile that generates DNS TXT records for a simple website