The Full Stack SysAdmin

I sometimes spend an unhealthy amount of time on Hacker News. I also communicate with a lot of people who work in Silicon Valley, or in a startup atmosphere. I've heavily noticed a trend towards talking about "DevOps" and "full stack developers", along with the not infrequent suggestion that IT professionals are no longer needed in the cloud era. (I've been told my profession is obsolete a few times.) Further than that, today I heard of the term "NoOps", suggesting that you really don't need anyone managing your company's technology assets at all.

To me, this is not only questionable from a factual standpoint, but this line of thinking causes people a poorer understanding of their business' needs, and how IT staff can fulfill those needs. Developers, I think, feel they can automate or abstract away IT with cloud services. On the other hand, one can also abstract away developers by purchasing software rather than developing it in-house. (Although, I wouldn't question the positive merits of having software perfectly-tailored to your business needs.)

First and foremost, for any business beyond a mom-and-pop shop, you require, and are paying for, IT. The question is how, and how far abstracted you are from it. For some, IT is just calling Geek Squad at Best Buy when a computer breaks. For others, they use cloud services, which means they're paying a subscription fee that includes IT staff they never talk to. In some cases, that subscription fee may actually pay for that service to pay a different service for IT. In fact, there's likely a profit layer there for each company involved, so you're paying quite a bit of overhead over the cost of that IT.

Now, there's no question that smaller companies may not need a full-time IT employee. That's why managed IT companies exist in the first place. Either by contract for part or full-time service, or on an as-needed basis with an incident fee. But I would contend that whether your IT is an in-house employee or a contractor, you should definitely know your SysAdmin.

The problem with the DevOps/NoOps concept, is that while looking for "full stack" developers, they miss the importance of "full stack" SysAdmins. By relying on the support of various service providers, you have no way of ensuring you are making the right decisions for your business, rather than the right decisions for your service providers.

Your Google Cloud support people cannot tell you if your business would benefit from using an Amazon Web Services product, or vCloud Air. They're there to support you using their products, rather than being their to support you.

IT personnel can not only support a particular component of your infrastructure, but understand your needs as a whole. For instance, on any given day I may be working with user desktops, repairing hardware on the plant floor, configuring virtual machines, running backups, managing accounts, pricing out and benchmarking competing technologies to our current choices, and responding to alert conditions on our web servers. If it's electronic, at bare minimum I can provide an informed opinion on it. I know my employer's workflow, and I can suggest better ways to do things, and implement the best options for the company. That's hardware, software, and networking alike.

So the question I have is, why would you trust your IT to someone who you'll never speak to directly? Who doesn't understand your priorities and the service levels you require? Who won't suggest solutions you didn't think of, or question why you're doing something in a less efficient, or less profitable, manner?

Google Glass: An Explorer's Postmortem

I think we're about at the point that I can comfortably talk about my experience with Google Glass as a whole from a holistic standpoint. Namely because as far as I am concerned, Glass is now dead.

Early Experience

In 2012, I was at Google I/O during the most public unveiling of Glass, and the first opportunity to get on the waitlist to buy one. Sergey Brin parachuted into the Moscone Center streaming video from the device, with a landing that felt like it was taken right out of Iron Man 2. I got GRID brick, number 663, and in May of 2013, I received the first of what ended up being three Shale Glass units.

I was also part of the Glass Foundry event the preceding January. I'll admit, one of the most exciting moments of my tech career was having a New York Times reporter beg me to break the non-disclosure agreement surrounding information we were told at the Foundry. (And there was a significant thrill in refusing to provide information to the New York Times. When newspapers want to know what you know... it's exciting.)

From my first experience with the device, I was absolutely thrilled with the hardware. It's elegant, simple, and blends right in, if you wear prescription frames and a muted color. The screen was perfectly designed; out of your way during daily life and easy to forget was there, but incredibly convenient to glance at when you needed it. Battery life was particularly short, but that is to be expected with a 560 mAh battery.

The first day I got Google Glass, I wore it to a restaurant and received, to my knowledge, the most amazing compliment a woman has ever given me. The girl at the register told me I was "the coolest guy she'd ever met" because I had Glass. I enjoyed the personal attention having one of the rare devices got me. One of my jokes was that "I paid $1,600 to buy popularity for a year." No longer having to seek people to converse with, people regularly approached me to ask about it.

The biggest initial concerns I had with the product was the design that all apps were simply messaging back and forth to the cloud via a Google API protocol (later to be supplemented by a native app option), and the fact that "Instant Upload" or Auto-Backup, was a non-configurable 'feature' of the device. If you took pictures with Glass, Google would upload those pictures next time the device had both power and Wi-Fi. I requested an option to disable this feature at the earliest possible opportunity.

Watching the Hype

As a moderator on the most popular Glass Explorer community on Google+, I had front row seats to the growth of the community and public perception of the Glass program. Invites were initially hotly-traded commodities, and people clamored for the chance to buy into the program. In time, Google granted their wish, first by giving Glass Explorers the chance to invite others to the program, and eventually by opening Glass up for public purchase.

Meanwhile, several people shifted their career focuses specifically to Glass-related products and services. Things like the SoGE (Society of Glass Enthusiasts) groups arose, along with companies making apps, skins, and hardware accessories for Glass. Decorative caps for the Glass earbuds were a particular favorite of mine. However, as someone not generally prone to the startup lifestyle, I questionned those who abandoned their other jobs to work an ecosystem surrounding a product that wasn't even released for consumers, as it seemed like an act of fanaticism. (Notably, some of these people have been hired by Google itself.)

Public Perception Sours

However, as excitement in the Glass community was ramping up, things turned sour for Glass in the public image. A few notable incidents in the San Francisco Bay Area were likely caused in part by the anti-tech movement that has arisen there. While Glass is far from a covert recording device, many people expressed believe that without a more overt recording indicator, Glass users were assumed to be violating their privacy. (Personally, I rarely if ever experienced negative perceptions while wearing Glass, most people I encountered were intrigued and curious about them.)

Furthermore, both businesses and the government turned against them. While I found the ability to subtly check notifications without disrupting moviegoers to be a primary perk of the device, movie theaters began to publicly announce bans on Glass for fear that people would use them to bootleg movies (despite obvious head-bobbing issues). The ability to navigate, text, and call hands-free on Glass is far superior to any hands-free system sold in any car today. However, states began to propose laws banning their use while driving, despite again, that being a key use case for the device.

While surely not more than a minor footnote, an affair between Sergey Brin and a Glass employee made a fair bit of a fuss when it went public, and shortly after even Sergey, our real-life Tony Stark, stopped wearing the device.

Glass Development Stalls

Another huge hit seemed to be the apparently slowing and eventual cessation of updates to Google Glass. While initially updated monthly, Glass eventually stopped receiving frequent updates. In particular the baseline upgrade to KitKat held back feature updates for months. One of the most publicly-marketed features, the ability to stream video live over Hangouts, was actually removed from the device, which was not a decision that was received well.

Very few Google apps were updated with more than the most basic functionality, even after a significant period of time. Settings were non-existant. For example, you could only receive emails Google decided were "important" and there was no way to change that behavior.

My longstanding privacy concern with Instant Upload was never addressed. For a while, it was arguable that Glass simply didn't have the depth of settings necessary to allow it, however Glass eventually introduced a settings bundle to configure auto-backup settings, allowing you to choose to allow sync regardless of power and network state, or to wait for wall power and Wi-Fi. However, Google refused to include an option to shut that synchronization feature off, showing disregard for users' ownership over their own photos.

In part due to the lack of consumer release, third party apps never left the range of around fifty apps, and the slow, Apple-esque app approval process meant many of the most useful apps never made it to the app list. Eventually, some companies dropped support for their Glass apps entirely. While the Android build on Glass was open source, and the bootloader was unlocked, the decision to keep the entire UI of the device in a closed source package killed the chance of any significant ROM development in it's cradle.

For many basic functions, no app was ever available. For instance, Glass never had an email client that could work with non-Gmail addresses. Without even this level of baseline capability, it was hard to justify wearing the device day in and day out, other than the attention it sometimes got me.

The End, Effectively

On a recent earnings call, Google confirmed that they asked the Glass team to step back and refocus. This was coupled with the closure of the Explorer Edition and the end of public sales. While Google still provides the devices for 'Glass for Work' customers, the consumer side of Glass has come to a close.

The official rhetoric is that they are working on what's next, I believe the end of Glass has been reached. Many of the known employees of Glass have now moved on, either by leaving the company or moving to other products at Google. I know of a few employees on the original Glass team which haven't, at least publicly, changed their status, but without so many of their developers, I question even the viability of the project continuing.

My Glass is now shelved, and I anticipate a matter of time for Google doing the same with the program.

The Android Trust: Why the EU Antitrust Case Matters

It's recently been revealed that the EU is preparing an antitrust case against Google. A lot of people have already started to come out and claim that it's crazy for whatever reasons. But it's truly one of the most important coming pieces of litigation. Android is a massive monopoly, and it was created by an agreement between a wide variety of competitors to work together.

The first thing to bear in mind, is Google is smart. Ridiculously smart. These are the people who social engineer the placement of food in employee cafeterias to trick their employees into eating healthier. Google knows what it's doing, and it knew what it was doing when it created the Open Handset Alliance (OHA).

When Android came out, most of the OEMs have no idea what to do about the iPhone. They didn't even have smartphone models yet in most cases. I refer to Android as the "open source trojan horse", because it appeared to be a gift at the time.

Google offered up an open source platform with the means to compete with Apple. OEMs were basically given free reign to do whatever they wanted with it, as long as they agreed to some basic provisions: To not break application compatibility. And, you know, Google was willing to help them and such if they installed the Android Market on it. At it's best point, Android was like Linux: A completely open source project that you could do what you wanted with.

But the problem is, the app ecosystem, whether people realized it at the time or not, is the most powerful component of the platform. Windows Phone is, in many ways, better than Android. But it doesn't have the app ecosystem, and without that, nobody wants one. (BlackBerry and Palm also learned this the hard way.) And as the Android Market, now the Play Store, of course, gained traction, it became less and less possible to sell a device without it. (Note that back then, a lot of OEMs did produce devices without the market. That practice has almost entirely died off now.)

With each Android version, the OHA agreements have been tweaked. Gradually bringing each OEM in line with Google's vision. OEMs don't want it, but they have no choice: They either fall in line, or lose the Play Store, and hence, their product won't sell. OEMs have fundamentally no control over the devices they produce at this point. There's strict limits on customization, strict limits on timelines OEMs must provide updates, strict limits on preinstalled apps. These are not Samsung, HTC, or LG phones. These are all Google phones.

One of the worst provisions is the all or nothing requirement with Google Apps. If OEMs want the Play Store, which they need, they also must include all of Google's other apps, like Google Search and Google+. Preinstalling Google apps makes it harder for other companies to compete on the platform. As Google no longer has to innovate to maintain hold over the market, and newcomers are disadvantaged, innovation languishes.

Technically speaking, no, OEMs aren't prevented from developing an alternative OS. But market forces have ensured that isn't a problem: They can't run the Play Store. The no forking provision coupled with the mandate to preinstall Google apps prevents OEMs from developing competitors to Google's apps and services without abandoning Google entirely. (There's a huge potential profit cliff to forking Android. Google knows it.) The few things OEMs are allowed to do by Google, are the things Google knows can't threaten them. Anything that can threaten Google's monopoly is regulated.

It's a combination of market forces and the agreements that they've made to fill in the gaps, that ensure Google maintains control over every Android OEM. Google is smart enough to have engineered these rules in a way that looks good on the outside, but is just as locked in nonetheless. Most of the companies making Androids have no control over their own products anymore, and are too afraid of the reprisals to break out from it. Samsung is probably the only company that has the brand power to break from Google successfully, but Google's been tossing them a bone here and there to keep buying them off.

A dissolution of the OHA would allow OEMs to differentiate their devices again, providing their own custom experiences and competitive products. Yes, more fragmentation would occur, but that's a good thing. Fragmentation is just another word for "consumer choice". And without the OHA, competing services would be able to compete with Google's built in services, encouraging new innovation in a market that has started to become stale.

The Divorce: I'm Leaving Google for Sandstorm

I've had a long relationship with Google. We've been together for nine years. I got a Gmail account in 2005. Since then, I've been a major user of almost every Google product ever released. I was a huge fan of Google Wave (rest in peace) and one of the earliest owners of Google Glass. Gmail introduced me distinctly to a company that was willing to put powerful features in the hands of users, and for quite a few years, Google seemed focused on a users first mentality. When I got my first smartphone in 2009, it was an Android, the Motorola Droid, the most customizeable device I'd ever owned. And my level of affection for Google only climbed.

But in the last couple of years, our relationship has soured. As Google defied industry expectations and began it's rise towards dominance, the focus seemed to change. No longer would the user be given control. Productivity began to take a back seat to style and design. The privacy policy and terms of use make for a one-sided relationship where only Google can dictate the terms of our interactions. A company that used to rely on best-in-class products to retain customers began to rely on walled gardens and platform lock-in.

Within the last year, I've come to the conclusion that I wanted a divorce.

My side of the marital assets: 6.4 GB of zip files, which make up my Google Takeout archive. Google keeps all the software I've been using for the last nine years, and data from several products which aren't a part of Google Takeout.

We live in an Internet-connected world. One where you expect to access your information online across a wide variety of devices. Anytime, anywhere. But currently this lifestyle requires that you hook up with one of a number of faceless corporations. Apple, Google, Microsoft, and others are all more than willing to house your digital lifestyle on the Internet... for a price. Sometimes it's money, but more often than not, it's your privacy. These corporations have discovered that your personal data is among the most valuable property you have, and also the property you are most willing to give away.

So I'm moving to, a new startup that I've been following. Sandstorm is developed by Kenton Varda, a former Google engineer, and a host of other team members and advisors, all with an incredibly strong experiential background. There's no doubt in my mind this group has the technical chops to pull off the impossible: Redesigning the Internet around people again.

Whether I choose to host my own server, or go with a major hosting provider, Sandstorm allows me to retain full control of my experience. I can use the web apps I want to, on the servers I want to. I can move those web apps anywhere I choose at a whim. Whereas leaving Google services means a painful transition to different software along with a data migration hassle, when I want to move from one Sandstorm server to another, I can pick up my app and go. Everything comes with me. And even if a developer abandons a web app and stops supporting it, like the graveyard of dead Google services, I can keep using it, because it's an app installed on my Sandstorm instance.

In case you weren't aware, you're actually viewing this blog on a Sandstorm server. In particular, this is Sandstorm's alpha server. I've set up several blogs before, and this is by far, the fastest blog setup I have ever done. Obviously the alpha server is temporary, and one day I'll need to move this blog to keep it online. But that is an incredibly simple process. Download it from this server, upload it to a new one, and update the DNS record. As I realize how much content I've contributed to Google+ since 2011, I realize that migrating that to a new server would be far more painful.

So from now on, I'm going to be mindful of where I put my data. My standing goal is to be entirely off Google services by December 31st, 2014. I'm going to put my data on things I can easily take and move elsewhere. I'm going to embrace standards and open source where possible.

Sandstorm is currently crowdfunding via an Indiegogo campaign and you can try it out on the demo server right now. You can also find it on Github.