You send a file, but how do you know it arrived intact? In other words, how do you know that it didn’t get cut off, garbled, or changed somehow? Simplistically, you could just add up all the bytes in the file — a checksum — and send that along with the file. You compute the checksum when you know the file is good, and the receiver can compare the checksum to see if they match.

However, a simple addition doesn’t catch certain classes of errors, which is why there are better checksum algorithms that, for example, wrap the carry bit around or otherwise modify files with common errors so they produce different checksums. There are two problems with checksums. First, no matter how much you modify the algorithm, the chances that two files produce the same checksum are pretty high. Especially with common error patterns.

For example, assume a very simple algorithm that simply adds the bytes and discards any carry. If a file contains 0x80, 0x80, those numbers essentially cancel each other out. If you replace them with 0, 0, you’ll get the same checksum. To some degree, using anything other than a second copy of the entire file will have this problem — some corruption goes undetected — but you want to minimize the number of times that happens.

The other problem is that a checksum by itself doesn’t let you correct anything. You know the data is bad, but you don’t know why. If you think about it, the simplest checksum is a parity bit on a byte: odd parity is simply summing all the bits together. If the parity bit doesn’t match, you know the byte is bad, but you don’t know why. Any even number of errors goes undetected, but I am sure one-, three-, five-, or seven-bit errors will get caught.

People invent better error-checking codes by devising schemes that can promise they can detect a certain number of bit flips and, at least in some cases, correct them. One of these is the cyclic redundancy check (CRC). It is easy to think of the CRC as a “strong checksum,” but it actually works differently. What’s more, there isn’t just a single CRC algorithm. You have to select or design a particular algorithm based on your needs. Most people pick a “named” implementation like CCITT or Ethernet and assume it must be the best. It probably isn’t.

A CRC is a checksum in the broad sense: you feed it a message, and it gives you a small value that you append, store, or compare later. But unlike a simple additive checksum, a CRC is based on polynomial division over GF(2), which is a fancy way of saying “divide using XOR instead of carries.” That detail matters. It gives CRCs very strong guarantees against common classes of errors, provided you choose the right polynomial for the job. That’s the key. You must choose the right polynomial.

The Polynomial Machine

A CRC treats your message as a long binary polynomial. For example, the byte stream is interpreted as a sequence of coefficients: each bit is either present or absent. The CRC algorithm divides the message polynomial, after shifting it by the CRC width, by a generator polynomial. The remainder is the CRC.

In normal arithmetic, division involves subtraction and carries. In CRC arithmetic, subtraction is XOR. That is why CRC code often looks like this:

if (crc & topbit)
crc = (crc << 1) ^ poly;
else
crc <<= 1;

That loop is implementing polynomial long division, one bit at a time. The generator polynomial is the magic number. For a 16-bit CRC, the polynomial has degree 16. For a 32-bit CRC, degree 32. You will usually see it written as a hex constant, such as 0x1021 for CRC-16/CCITT or 0x04C11DB7 for the classic Ethernet/ZIP/PNG CRC-32. But the polynomial is not just an arbitrary constant. It determines what error patterns the CRC is guaranteed to detect.

What CRCs Catch

A well-chosen CRC can guarantee detection of all single-bit errors, many multi-bit errors, all burst errors up to a certain length, and a very high percentage of longer random errors. The key metric is Hamming distance, often abbreviated HD. If a CRC has HD=4 for messages up to a certain length, it detects all 1-, 2-, and 3-bit errors in messages of that length.

That last qualifier is important. CRC strength is not just “16-bit CRC good, 32-bit CRC better.” It depends on the maximum message length. A polynomial that is excellent for 32-byte embedded packets may be mediocre for kilobyte-size messages. A polynomial standardized decades ago may be familiar but not optimal.

[Philip Koopman’s] work at Carnegie Mellon is the go-to reference here. [Koopman] and [Chakravarty’s] paper on CRC polynomial selection for embedded networks specifically looked for good CRC polynomials for short embedded messages, and [Koopman’s] “Best CRC Polynomials” tables list polynomials by width and Hamming-distance performance. The important takeaway is that many standard polynomials were chosen for historical reasons, not because they are mathematically best for your packet size.

There are plenty of videos that explain CRC, but if you are going to watch a video, you might as well pick one of the many from [Phil Koopman] himself, like the one below.

Famous Does Not Mean Optimal

Take CRC-16/CCITT, polynomial 0x1021. It is found everywhere: telecom, embedded examples, and bootloaders. It is not a terrible polynomial, but it is not automatically the best 16-bit choice. [Koopman’s] tables include other 16-bit polynomials with better Hamming-distance behavior over useful embedded-message lengths.

Likewise, classic CRC-32 using polynomial 0x04C11DB7 is deeply entrenched because of Ethernet, ZIP, gzip, and PNG. But CRC-32C, the Castagnoli polynomial, is often a better general-purpose choice. It has excellent error detection properties over common message lengths and is also supported by hardware instructions on many CPUs. Intel added CRC32 instructions with SSE4.2, and ARM AArch64 also includes CRC acceleration for CRC-32 and CRC-32C.

Of course, standards matter if you have to meet the standard. But if you are designing a new private protocol between your sensor board and your controller, blindly copying the first CRC-16 example from the Internet is not engineering. Pick a polynomial based on your packet length and your risk model.

The Practical Embedded View

For very small messages, even an 8-bit CRC may be adequate. For moderate packets, a good 16-bit CRC is often enough. For firmware images or large records, 32 bits is more reasonable. The point is not to use the biggest CRC you can tolerate. The point is to choose a CRC width and polynomial that give the desired detection strength for your longest protected message.

Also, remember what a CRC does not do. It is not cryptographic. It does not stop malicious tampering. The point of a CRC is to detect accidental corruption, not protect against sophisticated hacking attempts.

Real-world CRC definitions also include bit reflection, initial value, final XOR value, and sometimes byte order conventions. Two CRCs can use the same polynomial and still produce different answers because those parameters differ. That is a common embedded debugging trap. Someone says “CRC-16,” and both sides implement different CRC-16s. CRC-16/IBM, CRC-16/CCITT-FALSE, CRC-16/XMODEM, CRC-16/KERMIT, and CRC-16/MODBUS are not interchangeable.

If you specify a CRC in a protocol document, include at least the width, the polynomial (which can be represented in different formats, by the way), the initial value, if you use reflection on the input or output, and any value to XOR the output with. It is also a great idea to include the computed checksum for ASCII “123456789” so anyone can compare their algorithm to yours.

If you are working with Linux systems, be sure to look at the cksum program which can use several CRC algorithms or other methods like sha1 and other digest-style methods.

Efficiency

Computing CRCs a bit at a time is compact, but it costs eight loop iterations per byte. In some cases, that’s ok, but for performance, you want a table if you can afford the memory. For a 16-bit CRC, the table is only 512 bytes and can be generated at compile time, if desired.

Many CPUs have CRC peripherals. Use them, but read the fine print to make sure they can handle your desired CRC. It is often a good idea to check a hardware implementation against a known-good software implementation before you send it out into the wild. You can do many CRC tests using an online tool. Of course, there are several out there.

Choosing a CRC Today

For a new embedded protocol, define the maximum length of data you need to check. Then decide how many bits of overhead you can afford. Then head to Koopman’s tables to pick a polynomial with good Hamming-distance performance for that length.

The CRC has been around for a long time. But it isn’t just something you grab off the shelf. You need to plan and understand the ramifications of picking different polynomials.

CRCs aren’t the only game in town. Credit card numbers, for example, use check digits. There are other ways you can identify and, in some cases, zap bit errors, too.

Among so many other technological advances, the Cold War saw the advent of the ballistic missile submarine. The concept was simple—pack enough nuclear warheads to destroy a small civilization into a compact metal tube, and then hide it underwater. The oceans would act as a cloak for your fleet of world-enders, and keep your enemies forever on their toes. A terrifying machine that could both start and end a war with the push of a button.

Most nation states are populated by humans with the will to live. Thus, there has been a great incentive to find ways to keep tabs on these sunken doombringers. Great efforts have gone into improving sonar and magnetic detection methods over the decades, which are the bread and butter of sub hunting to this day. However, military researchers have also explored the prospect of whether submarines could be detected via their effect on the gravitational field alone.

Do You Feel It?

The simple matter is that every object with mass has its own gravitational field. We don’t typically think about it, because gravity is the weakest of the fundamental forces. On anything less than a planetary scale, it’s generally not obvious to us in our daily lives. However, submarines are quite heavy and large, particularly those that are armed with a complement of nuclear-capable ballistic missiles. Thus is raised the prospect of detecting these massive objects via their perturbations to the local gravitational field. This has been a hot-button news item in military commentary circles of late, with much bluster that advanced measurement equipment could potentially render the ocean transparent and reveal the locations of submarines at great distances.

Naturally, it’s difficult to comment accurately on top-secret military capabilities from a civilian viewpoint. Such a technology would be game-changing in a strategic sense, to the point that any nation state with such a capability would have great reason to keep its existence strictly hidden. However, there is some literature on the topic that is in the public domain, which discusses just how hard this feat would be to execute in practice. A great example is a report prepared by the Pacific-Sierra Research Corporation in 1989, under the sponsorship of the Naval Air Development Center.

How It Works

When it comes to detecting the gravitational anomaly of a submarine, you might think it would be easy given the sheer mass of such a craft. However, the way submarines operate frustrates this at a very fundamental level. In normal operation, a submarine is neutrally buoyant, displacing an amount of water roughly equal to its own mass. Thus, the submarine is not really distinguishable from the water around it in terms of its first-order effect on the gravitational field, being roughly as heavy as the water that would otherwise be there.

There is a wrinkle, though, in that a submarine is bottom-heavy for the sake of stability. This does create a variance in the gravitational field versus the otherwise uniform field in open water, and it’s one that could theoretically be detectable with a sensitive enough apparatus.

The device used for measuring gravitational variation is called a gravimeter. They are essentially a special-case variant of accelerometer, specifically designed to very accurately measure the local acceleration due to gravity at a single point. Then there is the gravity gradiometer, which measures the spatial rate of change of gravitational acceleration. By virtue of measuring acceleration gradients, a gradiometer is not sensitive to the acceleration perturbations of a moving platform, making it particularly useful for use in a moving frame of reference such as towing behind a ship or aircraft. Various types of each instrument exist, from portable units to high accuracy laboratory instruments; creating an exhaustive list of all  variants is outside the scope of this article. The real question is, based on the gravitational anomaly generated by a large submarine, to what useful range could a gravimeter or gradiometer detect one?

Unfortunately, the maths says that you have to get very, very close. In the 1989 study, calculations suggested the best gravimeters and gradiometers in the world would maybe be able to pick up a large submarine from a distance of tens of meters, at best. The simple problem being that the gravitational anomaly generated by an underwater submarine, and the gradient of that anomaly, are both so small, that even highly sensitive instruments would struggle to pick it up when the submarine is practically in visual range. Even if the problem were simplified, and one were trying to detect a submarine as a heavy point mass in empty space, detection ranges would stretch to somewhere in the range of 100 meters at most. Of course, this would be largely irrelevant due to the neutral buoyancy considerations explained above.

It’s true that technology has moved on since 1989. We have more advanced gravimeters and gradiometers available now, including quantum units with greater sensitivity than ever. And yet, even with these advances, it would be still be a struggle to detect a submarine at useful range. Sensitivities would have to jump by four or five orders of magnitude to enable detection at ranges of 1000 meters. Even still, if this were achieved with some highly classified system, it would still be relatively limited in capability versus more established techniques in magnetic or acoustic detection.

The parameters of the problem, combined with the sheer weakness of gravitational forces, means that gravitational detection is not some silver bullet for tracking enemy submarines at great range. While it would be desirable to have some kind of sensor that could reveal where these nuclear weapon platforms are lurking at all times, that technology seems beyond the reach of even the most capable navies at this time. For now, strategic planners will continue to sweat over the threat these weapons pose, never quite knowing whether they’re lurking just off the coast or half a world away.

If you were around tech in the bad old days, magnets could be really bad news. They were fine on the fridge, no problem at all. Put one near a floppy disk, or a hard drive, or even a computer monitor, though, and you were in for some pain. You’d lose data, possibly permanently destroy a disk or drive, or you’d get ugly smeary rainbow effects all over your screen.

The solid state revolution has eliminated a lot of these problems. We all use SSDs, flash drives, and LCD monitors now, all of which care a lot less about flirting with magnets. However, the same can’t be said about all our modern hardware, for a magnet could cause your smartphone some major grief indeed.

Magnetic Fields

As you might expect, the magnetic susceptibility of certain modern smartphones once again comes down to non-solid state parts. Now, there aren’t exactly a lot of phones out there that are packing hard drives or floppy drives or any sort of magnetic storage. Instead, it all comes down to cameras.

Take the modern iPhone line, for example. Apple is quite careful to warn against carelessly using magnetic accessories with the smartphone, because it can interfere with the cameras. Specifically, it’s because of the optical image stabilization (OIS) and closed-loop autofocus systems that are built into the cameras themselves. These devices use magnetic position sensors to determine lens position to compensate for focus, vibration, and movement, and use magnetic voice coil actuators to move optical elements, in order to take the best possible photos and videos at all times. If there’s a strong magnetic field in the vicinity of the lenses, it can interfere with this operation.

Few of us are sticking fridge magnets on our iPhones, to be sure. However, there are a lot of magnetic cases and mounts and other accessories that give people a great reason to stick magnets on their phone. In the cases of some third-party accessories that are poorly designed, it’s possible for these to cause problems with the camera if the magnets are too strong or too close to the key hardware. It’s worth noting that in typical use, something like a magnetic case or other small magnet won’t cause a lot of permanent harm. It will generally just degrade the operation of the camera until the magnet is removed.

This isn’t solely an iPhone problem, either. It can affect any phone that has any sort of magnetic sensing or actuation involved in the camera mechanism. Indeed, Samsung has even filed a patent on ways to mitigate this problem through carefully orientating the magnets used in folding phone mechanisms, and the appropriate use of shielding. Ultimately, similar camera technology is used in a great many phones, all of which are susceptible to this problem.

It’s true that in day to day use, you’re probably not going to run into a lot of problems waving around a magnet near your smartphone. Nor did floppy disks fail en masse in the 90’s, unless one of your colleagues was feeling vindictive and wiped them all with a fridge magnet on their lunch break. Still, like the oddball helium problem that because apparent with smartphones a few years ago, it’s funny to think that magnets could be causing trouble with computer hardware today. The fact is that a modern smartphone contains multitudes, and thus can surprise you with its edge case frailties.

Once upon a time, computing was simple. You had files on a floppy disk. If you wanted to take them to a different computer, you ejected the disk from one machine and put it in another. It wasn’t fast, but it was easy and intuitive. Besides, you probably only had one computer of your own, anyway.

Life has since gotten a lot more complex. You’ve got a desktop, a laptop, a work laptop, your personal and business phones, and a smart watch to boot. You live amongst a swirling maelstrom of terabytes of data. Despite all the technical advances that got you here, it’s still a pain to get a file from one device to another, even when they’re sitting on the same desk. Why?!

This Modern Glitch

Our computers are actually very good at connecting to each other. We have Ethernet devices with auto-negotiation, WiFi and Bluetooth in just about everything, and DHCP for good measure. It’s easy to get devices on the network and online. One might think all this connectivity would make sharing data easy. But we’re not so lucky.

Let’s take a straightforward example. Just getting a JPG off a smartphone requires jumping several hurdles and a little bit of begging to the benevolent tech gods. You can plug your phone in via USB to grab files, assuming you’ve got an Android, but you’ll have to flick through menus multiple times to get it to shift into the right mode to get files off. An iPhone will allow the same but you’ll need an app to help “import” them.

You could alternatively try sending them via Bluetooth, but you’ll have to go through the hassle of pairing, which almost never works first time. You’ll also get glacial transfer speeds and watching the process fail a few times. Alternatively you might see if your phone comes with a proprietary app for transfers, or you could try waiting to sync files to a cloud service or just emailing them to yourself. The latter method will make a mess of your inbox, but at least you get the files across when you need them.

It Was Not Ever Thus

It wasn’t always like this. Jump back a quarter century, and things looked very different. Windows 9x had a massive install base, with Windows XP just bursting on to the scene. You could still sneakernet stuff around with floppy disks if you wanted, of course. But it was also a cinch to set up simple network shares to access files across machines on a home network. It just worked.

Much the same was true of the Macintosh ecosystem. Back then, smartphones weren’t a thing, and few of us were carrying any sort of device with any real amount of data. Things like digital cameras and MP3 players would soon rise to prominence, but getting files on and off them was a dream—simply plug in, and they’d present as a USB mass storage device. No drivers, no passwords, no bloated apps. Just peace.

Of course, that would all change a few years down the line. Take the Windows world as an example. Network shares still exist, and you can set them up if that’s what you really want. Unfortunately, though, they’re so much worse than they used to be at the turn of the century. They’re buried under layers of permissions and user account nonsense that makes enabling them absolutely arcane. Only some of us run multi-user logins on individual machines, even fewer of us choose to run domain-style networks in our homes. In contrast, a lot of us would like it to be easy to pull a few files off the loungeroom computer when needed. However, doing so requires navigating passwords and accounts and setting permissions and if you get the slightest bit of it wrong, you won’t even see the shared files, let alone be able to access them. A task that used to take 3 minutes of setup now takes half an hour or more and a couple trips to Knowledgebase.

It shouldn’t be like this. One can imagine a world where all our devices in the home are allowed to share files openly and freely. Imagine if you could just click into the network tab on your PC, and see everything across all your devices – your laptops, your phones, your desktops and lab machines. Imagine not having to pair your phone or fiddle with utilities or special sharing tools or, god forbid, sending files all the way to the cloud just to move them three feet across your desk. Imagine this, all your files across all your machines at the click of a button, no auth, no nonsense, whether Apple, Windows, or Android. You already have all these devices talking on the same network, so all your stuff should just be there!

Alas, we cannot have such nice things. It’s not just because Big Tech is full of mean people that want to make life worse than it used to be, but it can feel that way sometimes. Instead, it’s more because of boring, sad, practicalities that are difficult to overcome. Security is perhaps the biggest headline issue in this regard. We now use our personal computers to store more private and confidential data than ever.. This makes access control paramount to avoid bad actors getting access to compromising information. There’s also the need to prevent the easy spread of viruses, which becomes very difficult when there’s a permissive file sharing route between devices. Malware has often taken advantage of holes in network sharing protocols as a vector for infection.

Beyond this, there’s the simple problem of interoperability. There isn’t a uniform standard that would allow easy, secure file sharing across laptops, desktops, and smartphones of all makes and models. This would require a large number of different tech companies to all get together, define a solution, and agree to implement it going forward. Sadly, current thinking seems to be that the proprietary solutions we have today are “good enough.” Apple’s AirDrop or Samsung’s Quick Share will get you by if you stay in the right walled garden, for example, and neither cares much to start a dialogue to establish something better and more cross-platform. Few tech companies would be excited about opening up potential security holes by implementing a new broadly-accessible file sharing protocol, either.

Perhaps a metaphor best explains the misery we find ourselves in today. If you live in a safe town with low crime, you might not feel the need to lock your car doors when you pop down to the supermarket. It means you can get in and out of your car without fishing for your keys, which is a great convenience when you’re carrying a bunch of heavy grocery bags. At the same time, you can’t live like this in a nastier place. Bad actors will simply open your door, rifle through your car, and take anything they like. That could end badly for you.

Unfortunately, cyberspace is that nasty place. By and large, we can’t just freely share files between devices because it’s too dangerous to do so. You don’t want your bank accounts drained, or your personal photos used for blackmail, so we have to drench everything in layers of authentication, even in the privacy of our own homes. Perhaps one day there will be some framework that allows us to create a close-knit network of “trusted” devices so we can freely move data about our own protected little bubble. But until then, we’ll have to suffer with Bluetooth passcodes and proprietary apps and the fact that it’s usually quicker to email a friend a photo then to find a way to directly transfer it to their phone which is sitting right next to you. It’s an annoying problem, and one that will not easily be solved.

When the World Wide Web surged into existence during the 1990s, we were introduced to the problem of how to actually find something in this ever-ballooning construction zone that easily outpaced even the fastest post-WW2 urban sprawl. Although domain names provided a way to find servers using DNS rather than having to mash in IP addresses, you still somehow had to know the relevant URL.

A range of solutions were thought up over time, ranging from printed Yellow Pages type guides, to online curated lists of resources, as well as things like web rings where one website would link to a relevant similar website. This was the time when word-of-mouth was also very relevant, with people proudly announcing their own website on Geocities or other hosting service.

Search engines already existed long before the WWW became the hot new thing during the 1990s, but it was the WWW that would really push them to their limits. As anyone who used search engines for the WWW can attest, they had many issues. Often you’d end up using multiple search engines to find something, and despite fierce competition between web search engines to become the starting page for their browser, actually finding things on the WWW remained a tough problem.

Since a web search engine ‘just’ has to index the WWW and match a search query against the results, why was this such a hard problem that persisted until Google apparently cracked the code?

Unplanned Sprawl

A nice thing about the WWW is that it was designed to be accessible to all, requiring only an Internet connection and thus opening up the possibility of setting up your own webserver. This unsurprisingly led to a very rapid growth of pages on the WWW, with content appearing, being modified and sometimes vanishing at an ever-increasing pace, making it extremely hard to keep up with.

This is however not how things started when the World Wide Web was created in 1989. Before its opening to the public in 1993 the pace of growth was slow enough that a manually maintained index was maintained. This was kept up until late 1992, with the last version of said index still online on the W3 website.

Over the course of a short few years, the WWW would change the face of the world forever alongside a surge of IBM-compatible PCs, exploding multimedia content, all the dot-com hype and perhaps best of all endless ‘free’ hosting services as long as you didn’t mind an advertising banner plastered above your personal homepage’s content.

Even internet service providers (ISPs) would often offer their own hosting service, along with endless n00b-friendly tools to make something resembling a website for whatever hobby you fancied. In addition to proving that one can absolutely argue about style and the prevalence of colorblindness, this would also serve to balloon the number of websites at an exponential rate.

Whether or not the WWW killing off the Gopher-based internet was a bad thing remains the topic of debate, though it’s beyond question that Gopher integrated search functionality into its protocol, mirroring a file system.

Infinite Library Indexing

Without any provisions in the HTTP protocol of the WWW, the only realistic way for search engines to create an index of the ever-expanding and changing WWW is to perform so-called web crawling. This means going through every known document, following any links found in them, and making sure to revisit any documents in case their contents got changed since the last visit.

The first complication here is that since the search engine’s database is the only real index for the web, initial discovery is purely organic, starting from a certain number of URL seeds in what is called the crawl frontier. This forms an integral part of a web crawler.

Development of the algorithms and architecture behind these crawlers formed a major part of the early WWW, with IBM researchers on the WebFountain project in 2001 estimating a grand total of about 500 million pages, with – as they put it – web crawlers caught between the comfortable cushion of Moore’s Law and the hard place of the web’s exponential growth. Today this number is probably closer to forty billion pages.

Although the Google Search web crawler was already pretty good back in 2001, WebFountain improved on it by using a distributed system, with ‘ants’ working through their own list of URLs to crawl, as described in the development paper by Jenny Edwards et al.

Beyond the basic recursive following of links in a document there are many confounding factors, such as when to recrawl a URL, which very much depends on how often the content on it is expected to be updated. Here one dives into the territory of statistics, as depending on the type of site we can make an educated guess on how often it is expected to be updated. For example, a government’s historical news pages are unlikely to see frequent updates, whereas the front page of a news site can see updates practically every few minutes.

Inverted Indexing

As complex the topic of web crawling is, the fun part begins when you have pruned all duplicate documents and stripped all the irrelevant fluff that’s not text to be indexed. In order to make the resulting search index at all searchable before the heat death of the Universe you cannot simply do a full text search on every single document whenever someone enters a search query.

Instead an index is constructed whereby certain keywords are mapped to documents. This inverted index is generally implemented as a hash table or similar data structure where it provides a quick access into the full text documents, not unlike the keyword index in the back of a book, or the more elaborate concordance of yesteryear. These latter works also provide a keyword index, but add accompanying text to provide immediate context to further save time.

Creating an inverted index is a fairly labor-intensive process, with a new document often used for a forward index that decomposes the text into its keywords prior to updating (or creating) the inverted index. As with all of such text processing related tasks and data structures in general there are many ways to go about it, with some fun curveballs thrown into the mix such as parsing languages that do not separate words with spaces, like Japanese.

All of which is to say that implementing a search engine is easy, but making it performant, accurate and efficient  at the same time is a minor nightmare. This is basically why search engines took so long to stop being so terrible, as the engineers behind them were trying to solve many rather complex problems, presumably with the C-suite and investors breathing down their necks during the dot-com days.

Search Battles

Over on the Wikipedia entry for ‘Search engine‘ we find a pretty good timeline of web search engines, along with their current status. Perhaps unsurprisingly none of the 1993-era ones made it, but 1994’s WebCrawler somehow crawled into the modern age, along with Lycos. Much like 1990’s Archie search engine and similar for the Gopher web, many of these early search engines simply couldn’t compete in the rapidly changing years leading up to the new millennium.

This was also the era in which some figured that the WWW simply needed to become more ‘3D’ with virtual environments using VRML, bringing it closer to sci-fi like that portrayed in Snow Crash or Tron. Perhaps unfortunately the WWW remained the domain of mostly text and images, although most recently the flood of JavaScript frameworks appear to want to turn once simple HTML documents into full-blown desktop-like applications, all probably to the delight of web crawler engineers.

Meanwhile some search engines figured that they could lift along on the hard work of others, with so-called meta search engines collating the results from multiple search engines to save people the trouble of querying them individually. Here 1996’s Dogpile is still going strong.

Some search engines are missing from the list, such as Marginalia, which boasts the use of open source software for its indexing and crawling, while focusing on non-commercial content. There is also the ever excellent Frog Find that provides a bridge between modern search engines and systems that really cannot run the latest web browser.

Today’s Survivors

The search engine landscape remains a brutal one today, with us having to recently say farewell to Jeeves, of Ask Jeeves fame, most recently seen carrying the Ask.com name. Personally I didn’t really Ask Jeeves much back in the day, instead mostly using AltaVista (RIP) and probably Lycos and a few others that I do not recall off the top of my head.

Having Google Search burst on the scene by 2000 was definitely quite the event, which was certainly when the web search game improved. Looking back it probably was less that Google Search was simply better, but more that it pushed hard just being a search engine, whereas the others were still very much stuck in that early WWW mindset of being a portal to the web.

To a certain extent this is understandable, as search engines aren’t a charity and running the associated hardware as well as the required bandwidth costs a lot of money. Despite this it would seem that we still have a rather thriving web search engine landscape, even if ChatGPT, Claude and kin are trying to become the very last ‘site’ you will ever need. This even as their little web crawlers are still doing the same crawling as has been done since the birth of the WWW.

When most people think about vacuum tubes, they picture big glass bottles glowing inside antique radios or early computers. History often treats tubes as a dead-end technology that was suddenly swept away by the transistor in the 1950s. But the reality is much more interesting. Vacuum tube technology did not simply stop evolving when the transistor appeared. In fact, some of the most sophisticated and technically impressive tube designs emerged after the transistor had already been invented.

During the final decades of mainstream tube development, manufacturers pushed the technology in remarkable directions. Tubes became smaller, faster, quieter, more rugged, and more specialized. Designers experimented with exotic geometries, ceramic construction, metal envelopes, ultra-high-frequency operation, and even hybrid tube-semiconductor systems. Devices such as acorn tubes, lighthouse tubes, compactrons, and nuvistors represented a last gasp of thermionic electronics.

Ironically, many of these innovations arrived just as solid-state electronics were becoming commercially practical. Vacuum tubes were improving rapidly right up until the market abandoned them.

The Pressure to Improve

By the 1930s and 1940s, vacuum tubes dominated electronics. Radios, radar systems, military communications, industrial controls, and the first digital computers all depended on them. But everyone was painfully aware of their problems.

Traditional tubes were fragile, generated heat, consumed significant power, and suffered from limitations at high frequencies. Internal lead lengths created parasitic inductance and capacitance. At radio frequencies and especially microwave frequencies, those unwanted effects made design difficult.

Military requirements during World War II accelerated development dramatically. Radar systems needed tubes capable of operating at VHF, UHF, and microwave frequencies. Vehicle equipment required devices that could withstand punishment. Computers with tubes suffered from frequent failures, took up entire rooms, and needed special cooling equipment, often bigger than the computer. These pressures drove tube designers into an intense period of innovation.

Acorn Tubes: Tiny Tubes for High Frequencies

One of the earliest major departures from conventional tube geometry was the acorn tube. Developed in the 1930s by RCA, the acorn tube got its name from its distinctive shape, which resembled an acorn with wire leads protruding from the base and sides. Unlike ordinary tubes, where the internal elements had relatively long leads, the acorn design minimized lead length to reduce parasitic capacitance and inductance. At high frequencies, this reduction was crucial.

One famous example was the 955 acorn triode. These tubes found use in experimental television receivers, military radios, and laboratory equipment.  Acorn tubes also reflected an important trend in late tube development: engineers were increasingly treating tubes not merely as amplifying devices, but as microwave structures requiring careful electromagnetic design.

The Lighthouse Tube

If acorn tubes were specialized, lighthouse tubes were positively futuristic. Lighthouse tubes abandoned the classic cylindrical glass form almost entirely. Instead, they used stacked disk-like electrodes arranged in a compact coaxial structure. The resulting geometry minimized transit times and parasitic reactances, allowing operation into microwave frequencies.

The tubes vaguely resembled a lighthouse tower. These tubes became essential in radar systems during World War II and the early Cold War period. Some lighthouse designs could operate in the gigahertz range, something impossible for conventional receiving tubes.

Their construction also introduced new manufacturing techniques. Many used ceramic and metal rather than large glass envelopes. This improved heat resistance and mechanical stability while reducing losses at high frequencies.
In many ways, lighthouse tubes represented the transition from classic vacuum tubes and true microwave devices like klystrons and traveling-wave tubes.

Metal Tubes and Ruggedization

Another path of tube evolution focused on durability and compactness. Early tubes used fragile glass envelopes that were easily broken and susceptible to microphonics and vibration. During the 1930s, manufacturers introduced all-metal tube designs. These tubes replaced the glass envelope with a metal shell, improving shielding and mechanical ruggedness.

Metal tubes were particularly attractive for military and automotive applications. Shielding reduced interference, while the smaller physical size allowed more compact equipment layouts.

Hybrid glass-metal constructions also became common. Engineers experimented constantly with new materials and packaging approaches to reduce noise, improve reliability, and extend tube lifespan.

Subminiature Tubes

One of the most impressive developments was the subminiature tube. These tiny devices often looked more like oversized resistors than conventional tubes. Some were less than an inch long and designed to be soldered directly into circuits rather than plugged into sockets.

Subminiature tubes emerged largely from military demands during and after World War II. Proximity fuzes for artillery shells required electronics small enough to survive being fired from a cannon. Traditional tubes would simply shatter under the acceleration.

The resulting ruggedized miniature tubes were shock-resistant and compact enough for portable military electronics. After the war, subminiature tubes appeared in hearing aids, portable radios, test instruments, and early miniaturized computers.

The Nuvistor: The Ultimate Receiving Tube

One of the most interesting late-stage vacuum tube was the RCA Nuvistor. Introduced by RCA in 1959, the nuvistor represented an attempt to create a truly modern vacuum tube for the transistor age.

Unlike classic glass tubes, nuvistors used a compact metal-and-ceramic construction. They were extremely small, highly reliable, vibration-resistant, and capable of excellent high-frequency performance. They also exhibited very low noise characteristics. At first glance, a nuvistor hardly resembles a traditional tube at all. You could easily mistake these for some other component in a metal can.

Technically, nuvistors were excellent devices. They offered superior performance in many RF applications compared to early transistors, particularly in television tuners, instrumentation, and aerospace electronics.

High-end studio microphones also adopted nuvistors because of their low noise and desirable electrical behavior. Some audiophiles still use nuvistor-based equipment today.

But despite their capabilities, nuvistors arrived too late. Semiconductor technology was improving rapidly. Silicon transistors were becoming cheaper, more reliable, and easier to manufacture in large quantities. Integrated circuits loomed on the horizon. The nuvistor may have been the best small receiving tube ever made, but it was competing against a technology whose economics would soon become overwhelming.

Compactrons

As semiconductor electronics advanced, tube manufacturers attempted another strategy: integration. The Compactron, introduced by General Electric in the early 1960s, combined multiple tube functions into a single envelope. A compactron might contain several triodes, pentodes, or diode sections in one package. This reduced component count, simplified wiring, and lowered manufacturing costs for television sets and other consumer electronics. Of course, tubes with multiple electrodes weren’t new. They dated back to at least 1926. However, GE’s aggressive marketing of the brand was an attempt to prevent designers from defecting to the solid-state camp.

In some sense, compactrons were the vacuum tube answer to integrated circuits. Engineers were trying to achieve greater functional density while keeping tube-based designs economically competitive. GE’s Porta-Color, the first portable color television, used 13 tubes, including 10 Compactrons. They usually have 12-pin bases and an evacuation tip at the bottom of the tube rather than at the top.

Compactrons saw widespread use in televisions, stereos, and industrial electronics during the 1960s and early 1970s. But again, semiconductor integration advanced even faster. The battle was becoming impossible to win.

Specialized Tubes Survived

Even after transistors took over consumer electronics, vacuum tubes remained important in specialized fields. Microwave tubes such as klystrons, magnetrons, and traveling-wave tubes continued to dominate high-power RF applications. Radar systems, satellite communications, particle accelerators, and broadcast transmitters all relied on advanced vacuum devices. In some areas, they still do.

A modern microwave transmitter aboard a communications satellite may still use a traveling-wave tube amplifier because tubes can handle very high frequencies and power levels efficiently.

No Instant Win

One misconception about electronics history is that the transistor immediately rendered tubes obsolete after its invention at Bell Labs in 1947. That is not what happened.

Early transistors had many limitations. They were noisy, temperature-sensitive, low-power, and expensive. Tubes often outperformed them in RF circuits, audio applications, and high-power systems well into the 1960s.

For a significant period, designers genuinely did not know which technology would dominate certain markets. Tube designers were still making substantial advances. Nuvistors and Compactrons were not desperate relics; they were serious engineering efforts intended to compete in a changing world.

Ultimately, however, semiconductors possessed overwhelming long-term advantages. Transistors required less power, generated less heat, occupied less space, and could be manufactured using scalable photolithographic processes. Once integrated circuits became practical, the economics shifted decisively. Vacuum tubes could evolve, but they could not shrink into millions of devices on a silicon chip.

The final years of vacuum tube development are often overlooked because history tends to focus on winners. Yet this period produced some of the most elegant and specialized electronic devices ever created. By the late tube era, vacuum tube manufacturing had become quite refined. Engineers could produce tubes with tightly controlled characteristics and surprisingly long operating lives.

Some early transistorized devices still retained subminiature tubes in certain high-frequency or low-noise stages because transistors had not yet surpassed tube performance in every application. This overlap period is often forgotten today. Electronics did not instantly switch from tubes to semiconductors. For years, many systems used both. For many years, a typical ham radio transmitter, for example, would be all solid-state except for the power amplifier finals, which were often a pair of 6146 tubes.

You can, of course, make your own tubes. If you’ve had enough of making your own tubes, maybe try reproducing some of these advanced models.

The Global Positioning System (GPS) was developed by the United States military in the 1970s, but it wasn’t long before civilians all over the planet started using it. By the early 2000s the technology was popping up in consumer devices such as mobile phones, and since then its become absolutely integral to our modern way of life.

But although support for GPS in our gadgets is nearly ubiquitous, it’s not the only option when it comes to figuring out where you are on the globe. As you might imagine, not everyone was thrilled with building their infrastructure around one of Uncle Sam’s pet projects, and so today there are several homegrown regional and global satellite navigation systems in operation.

As a follow-up to our recent dive into the ongoing GPS upgrades, let’s take a look at some of the other satellite positioning systems and who operates them.

GLONASS (Russia)

Given the tensions of the Cold War, it will probably come as little surprise to learn that the Soviet Union introduced their own satellite-based navigation system to compete with GPS. Development of the Global Navigation Satellite System (GLONASS) started a few years later than its Western counterpart, with the first satellites not reaching orbit until 1982, officially making it the second Global Navigation Satellite Systems (GNSS) ever developed.

GLONASS satellites orbit at a slightly lower altitude than GPS, 19,100 kilometers (11,900 miles) compared to 20,200 km (12,600 mi) of the American system, and at a greater inclination. This makes reception better at higher latitudes, which makes sense given the desired coverage area.

As designed the capabilities and overall accuracy of GLONASS were very similar to GPS, but the early satellites had a short operational lifespan of just three years. For global coverage GLONASS required 24 satellites in orbit, and maintaining coverage over Russia required 18. But after the fall of the USSR, launches of new satellites were put on pause and the constellation started suffering losses. By 2001, there were just seven operational GLONASS satellites.

President Vladimir Putin made the restoration of GLONASS a key priority in his administration, leading to resumed launches and development of the second and third generation satellites. Within a few years, commercial interest in GLONASS started to pick up, and the network regained global coverage in 2011. While the constellation has experienced a few setbacks over the last several years, spare and replacement satellites have been launched regularly, with the most recent entering orbit in September of 2025.

BeiDou (China)

Unlike the American and Russian systems, the first iteration of BeiDou was of a much smaller scale. Rather than a global system, the goal was to provide regional coverage for China and the surrounding countries with just four satellites placed in a geostationary orbit at an altitude of approximately 35,786 km (22,236 mi). From an observer in China, the satellites would appear to be motionless in the sky, ensuring reception anywhere in the country. Known retroactively as BeiDou-1, the system was operational from 2003 to 2012.

That year it was replaced with the far more ambitious BeiDou-2. The design called for a constellation of satellites in various orbits: 5 geostationary to provide backwards compatibility with BeiDou-1, 27 in medium Earth orbit similar to GPS/GLONASS, and 3 in an inclined geosynchronous orbit. The latter meaning that from the perspective of Earth, the satellite would appear to loiter overhead rather than remain in a fixed position.

BeiDou-1 was largely a research project and saw little use outside of the Chinese government. Conversely BeiDou-2 was designed for both government and civilian use from the start, with two distinct levels of service — civilian users could plot their position within a radius of 10 meters (32 feet), while the military reportedly enjoyed an accuracy of 10 cm (4 inches).

The coverage area of BeiDou-2 was expanded considerably to the south to include include Indonesia and Australia, but it still didn’t provide global service. Commercial use of the network started to pick up at this point, and by 2014 smartphones from Sony, Samsung, and Xiaomi included support for it.

It wasn’t until the introduction of BeiDou-3 in 2015 that the system could boast global coverage, with the system reaching full operational status in June of 2020.

Galileo (European Union)

While civilian use of GPS, GLONASS, and BeiDou was always part of the plan, all three systems were ultimately designed  as tools of their respective governments. Conversely, when the European Union set out to develop Galileo in the early 2000s, the goal was to create a satellite navigation system operated by private companies and aimed at civilian users.

That first part of the plan fell apart fairly quickly, and by 2006 Galileo was nationalized and the European Space Agency was entrusted with its development and operation. The first operational satellite was put into orbit in October 2011, and limited functionality was available to the public by 2016. While Galileo was designed for civilian use, it does offer a High Accuracy Service (HAS) with an accuracy of 20 cm (8 inches) that was initially intended to be accessible only by paying customers. But eventually it was decided to make HAS available to compatible receivers free of charge. When combined with its interoperability with GPS and GLONASS, Galileo offers exceptional accuracy.

Galileo reached full operational status in 2024 with a constellation of 24 satellites. Starting in 2027, these will be joined by a dozen upgraded Galileo Second Generation (G2) satellites that feature more electric propulsion for more efficient orbital maneuvers, improved antennas, and inter-satellite data links.

QZSS (Japan)

Development of the Quasi-Zenith Satellite System (QZSS) started in 2002, with the goal of offering high-accuracy position services to users in and around Japan. But rather than operating independently, QZSS was designed to augment GPS with five additional satellites.

Two of the satellites are in a geostationary orbit similar to those used in China’s BeiDou-1 system, while the other three are in a geosynchronous orbit like those introduced with BeiDou-2. These orbits are intended to keep at least one satellite directly over Japan at all times to improve reception in urban areas. The system became fully operational in 2018.

In the near future, Japan plans on adding three more satellites to the QZSS constellation. This would give the system enough regional coverage to operate independently of GPS if necessary.

NavIC (India)

Navigation with Indian Constellation (NavIC), previously known as Indian Regional Navigation Satellite System (IRNSS), is an independent regional navigation system that covers India and the surrounding area using seven satellites.

Development of NavIC started in 2006, and the first satellite was launched in 2013. Like QZSS, the constellation is made up of satellites in both geostationary and geosynchronous orbits. Two levels of service are offered: the Standard Positioning Service for civilian use that offers an accuracy of 3 m (9.8 feet), and an encrypted Restricted Service intended for military and government applications that’s accurate to 2 m (6.7 ft)

One of the goals of NavIC was not only to launch and operate the system from within India, but to produce as much of the hardware domestically as possible. This includes the atomic clocks and microprocessors aboard each satellite as well as the receiver chips used in client devices. While India wanted to maintain ultimate control over NavIC for political reasons, it’s not an isolationist system — it is designed to be interoperable with other GNSS.

That last point is particularly important right now, as only three NavIC satellites are currently transmitting navigational data due to hardware issues. Those three satellites alone aren’t enough to plot an accurate position, so to compute their location receivers must pull in data from other systems such as GPS.

Better Together

Although having so many active satellite navigation systems may seem redundant, the fact that they all offer at least some level of interoperability with each other means that everyone with a multi-system receiver can benefit. Instead of being limited to the constellation of just one service, this cross compatibility lets a device pull in data from whatever satellites are overhead at the time.

Granted how much of an improvement this results in will be highly dependent on where you’re located on the globe, but no matter what, its always going to be better than being limited to just one system.