Ep. #4, Getting Down To The Metal

Guy Podjarny: So hello everybody. Welcome back to The Secure Developer, where we talk about everything security for developers, including tools, best practices, and general ideas about how you can help build more secure applications.

With me today, I have the pleasure of having Eric Lawrence from Google, who's doing developer relations that focus specifically on security. Thanks for coming on the show, Eric.

Eric Lawrence: Great to be here.

Guy: So, I've got a whole bunch of questions for you, but before we dig into that, can you give a bit of background about your job today, how you got into security, what's the path that led you to your spot today in the security developer relations world in Google.

Eric: Great, sure. So I'm actually, technically, I'm a software engineer on the Google Chrome security team; but I got hired for the purpose of evangelizing HTTPS to the masses, and finding places where the Chrome team could do more to help developers move seamlessly to HTTPS, and so my life right now is pretty much all about getting sites onto secure protocols and helping them out in that task, wherever we can.

Whether that's by creating documentation, guidance, making changes in Chrome to smooth rollouts, or talking to the developers who are encountering problems and figuring what those problems are and finding a way to help them out. How I got there is a bit of an odd path. I know a lot of people sort of start out in security from the beginning with the notion of, "Hey, I want to do software security, so so cool."

I'm something of an old guy, however; I was an intern at Microsoft in 1999, and I worked on the product that turned into SharePoint. At the time, it was called Office Webserver. I've been doing web things for quite a long time, but I had no particular security bent. Just before I started full-time at Microsoft, in 2001, I was on the phone with my new team, who hadn't worked with me previously. I was going to work on a new feature team within Office called Assistants In Worldwide Services, and my conversation with those guys was interrupted by my computer, that played a clip from Star Trek. Something about, you know, 'incoming fire, shields holding.' The person on the phone asked what that was, and I laughed.

What it had been was, at the time, this was back in the heyday of IIS worms, and so there was a worm going around campus at the time, attacking Windows servers, Windows 2000 servers; and I had written a little ISAPI filter. It was a trivial little thing that looked at the incoming queries coming in to the web server, and if it saw the signature of a known attack, it would block it and it would play this audio clip.

The guy that I was talking to on the phone said, "Oh, that's great. You've got a background in security: you'll be our security PM." And I laughed really hard, and assumed he was joking. Then I showed up on the first day, and asked what I was working on, and they said, "Well, you know you're going to own our clip art website." The clip art website at the time had about a million visitors a day, and so that was kind of cool. And they said, "of course, you'll also be our security PM."

I laughed, and tried to explain, "No, no, I don't know anything about security." The response was effectively, "Well, no one else here does, either" because this was just the beginning of the trustworthy computing timeline at Microsoft.

"We have this book from Michael Howard, Writing Secure Code, so you'll read that, and you'll become an expert on that topic."

I thought this was a little bit preposterous; but, I'm the new guy. I'm not really going to protest too much that I can't do a job that they've hired me for. And so I read the book, and it turned out that one attribute that I have was really super useful for security; and that's curiosity about how things work, and a willingness to go all the way down into the metal.

One of the things that I did, over the following year, was really get into security things.

And you don't think of clip art as a good place to get into security; but as it turns out, the clip art file formats were super interesting because everybody trusts clip art. It's just clip art, right?

As a consequence, all of those file formats were configured to automatically open on Windows machines with no prompt whatsoever, and there was no gating as to where they came from.

You could be surfing the web, and the site could serve you a clip art package, which would immediately be opened by native code on your machine with no prompts whatsoever. I spent a summer taking over a machine with clip art, and I thought, "Hey, I've actually got something of an aptitude for this." That's kind of how I got into security, and kind of carried me over a little bit.

After a couple of years of working on clip art, I joined the IE team: not because I wanted to work on security, but because there were some problems in the way IE6 was behaving in handling our clip art website, and Microsoft source code wasn't open across teams. So I figured, "Well, I can join the IE team, find out all the bugs that are causing this problem for our website, and fix them."

I did that, but as a consequence, I was also on the Trusted Networking Team, and so effectively I started working for security teams in IE starting in 2004. After about, I guess it was four or five years, I started leading the security team for IE, and so I did that for a couple of years.

On the side, I was working on a side project called Fiddler. Fiddler is a web debugging proxy. It runs on Windows and on Linux, now. It'll kind of sort of on Mac, but not really. Fiddler, being an HTTP proxy, is in a very privileged position to do things that are very interesting for security and security testers.

I got kind of deeper into security through Fiddler, and in 2012, I left Microsoft when Fiddler was acquired by Telerik; and I moved to Texas, of all places, and started coding Fiddler for Telerik. I did that for three years.

When an opportunity came up to join the Chrome security team, I really was impressed by the quality of the people that Chrome had working on security, and was excited at the opportunity to do security in web browsers again for a while.

I joined the Chrome security team, and started on January of 2016.

Guy: Cool. That's quite a journey. Actually, a lot of interesting things about it. One, the thing you noted about having some aptitude for security being related to the curiosity to go all the way deep down: I feel like that's oftentimes also a good property of a good developer, of somebody that wants to understand the software, sometimes debugging, or troubleshooting an ops problem, or something like that.

It's very much this problem-solving case where you need to get down and understand the bits and bytes of what's moving and what's happening, and I like that. I always feel like there's a lot of separation sometimes.

In the world of security, you talk about the breakers and the builders; and how a breaker's mind, an attacker's mind works so differently, and that it's a different person, different persona, and I don't really subscribe to that proposition.

I feel like the developer mindset of understanding the components, maybe the engineer mindset, actually very much applies to security as well; you anticipate bugs versus trying to build around them. I like that, and I'm definitely a fan of Fiddler. I've used it a fair bit, until I moved to a Mac, which unfortunately killed that opportunity. As long as I was working on Windows, Fiddler was very much a favorite tool. I appreciate that, I'm a happy user of the tool.

Eric: Thanks!

Guy: So I guess, fast-forwarding from that path, right? From all this world, you sort of landed in the Chrome security team, and I think today this world of browser security, of front-end security, right? This sort of term that's not quite well-defined yet is an interesting one.

It seems to have evolved. I think a lot of security controls have been added to browsers today to go beyond just the notion of securing the browser itself, on to providing controls to make the application, the web application, that is running through the browser, secure.

How do you see those right now? Do you have some favorites of the recent additions? Is there some some trend that, at least from the Chrome perspective, you're trying to achieve?

Eric: I've worked in browsers for quite some time now: it's 12 years, which seems utterly ridiculous to me, because it all feels so new, and that's partly because things are changing so much. In 2004, when I joined IE, all the interest was effectively, "Hey, we need to get root on the machine."

So you get an arbitrary code execution through some memory corruption, and you completely own the machine. And there was a ton of investment in things like sandboxing the browser, and making it hard to get reliable memory corruption and code execution. Things like DEP/NX and ASLR, things to complicate the allocation of memory such that it was less predictable.

Those things were really the focus of what the IE team was working on in the 2004 to 2007-ish time frame. Eventually Sandboxes got reasonably good, particularly the Chrome Sandbox, and memory protections and other features got better as well. The challenge sort of moved a little bit, because in the old days, the reason that we wanted to protect the machine so much was that's where the user kept all their information; and so if you got remote code execution on the machine, you needed it in order to steal the information that the user had on the machine.

The world has moved quite a bit since then, and for many people, all of the data that's of value to them is in the cloud somewhere. This is particularly true on devices like Chromebooks, where there's very little local storage, and almost everything is in the cloud; but even people who are using Windows PCs or Macs are very often, all the data that matters to them is in the cloud.

You've got this situation where, certainly bad guys are still very interested in getting arbitrary code execution on a local machine, because if they can own your local machine, they can attack your cloud. But you don't need local code execution, necessarily, anymore to attack the cloud; and this is the notion of where web app security comes into play, this idea that

As more and more of your data moves to the web, that's where the protection needs to be.

Something like a universal cross-site scripting bug that allows a site to steal data from any other website you happen to be logged into is now, in many cases, as dangerous as a remote code execution bug in your browser itself. We've gotten a lot more interest now in providing ways for web applications to defend themselves against attackers.

Coupled with that, we also have an increase in capabilities of the web platform. Back in the day, if you had arbitrary code execution in the user's browser that was confined to the browser within the Sandbox, there wasn't necessarily a whole lot you could do, because we didn't have APIs, necessarily, for things like recording a user's camera, or turning on a user's microphone, getting their location, and so forth.

Now, all those capabilities are moving to the web platform, and that's because the web platform needs these capabilities if it wants to be a compelling platform for developers who have the option of doing things like building mobile applications in Java for Android, or in Objective-C or Swift for iOS.

The web platform is inherently getting more powerful, and thus, you know, a compromise that allows you to execute code certainly is something that is much more interesting than it has been in the past. We've got a lot of browser-side features: some of the first ones were things, very trivial things like HTTP Only cookies and cross-site scripting filters and things like that.

Over time, things evolved a bit, with attacks like clickjacking, or UI redress attacks, as they like to call them in the security research community, where the user is enticed to perform an action in their browser where they don't recognize that it's happening. Browsers have started to layer on defenses against specific attacks like that; as well as starting to create primitives that are of interest, to have analog in the old world, and now are applicable to the web.

You've got things like Subresource Integrity hashes, so you can say, "Hey, go download this shared JavaScript library from this third-party CDN site, but only run it and allow it to execute code if the hash is the thing that I'm expecting it to be." That way,

If I happen to compromise some CDN's repository of jQuery, I don't get to own the entire internet; because sites that are using Subresource Integrity will not run those JavaScripts

because they're not the scripts that the developer expected.

We're starting to see things like that. We're also starting to see some gradual introductions of things to restrict legacy features that were not designed in a way that we would have designed them today. In particular, you've got HTTP cookies. Cookies are super useful, and they're used by virtually every site, but cookies have some properties that make them very bad from a security point of view, things like there's not a good distinction, necessarily, between cookies that came from a secure origin versus cookies that came from an insecure origin.

Many sites are both HTTP and HTTPS, and if you can get a cookie set on HTTP, that cookie will get sent to the HTTPS server; and so cookies didn't have a protection against that. There's a protection going the other way: you can mark your cookies secure, so that a securely-set cookie is not sent to HTTP, but it doesn't prevent the reverse.

Now, there's actually a feature called Cookie Prefixes, which is working its way through the design process, where a cookie can have a magic name: if it's prefixed with __host or __secure, it's not set unless it was set from a secure origin. And so, a server, upon receiving such a cookie can tell, "Hey, this was set securely, so I know it came from me."

A lot of the security that we're designing today is predicated on the notion that the user is getting the application from the server securely; because

As you add these powerful capabilities, it becomes very dangerous if there's anyone on the network that can interfere with the delivery of that application.

In the same way that when you downloaded your programs to run natively on your PC, you're trusting that Windows or Mac is actually checking the signature of that program to make sure that that program is the one that you asked for; it wasn't corrupted by a virus or tampered with by a third party.

Well, HTTPS is really what we have for the web, and so a lot of the capabilities that have been added to the web platform are starting to require HTTPS; and so if you want to use geolocation, you need to deliver your website over HTTPS connection or the geolocation calls will fail.

The thing that Google, as a whole, is probably most excited about from the web platform perspective is what we're calling Progressive Web Applications. These are web applications that start to blur the line between the web applications of old and the native applications that have become so popular for mobile.

The most powerful feature of Progressive Web Applications is called ServiceWorker. ServiceWorker is kind of like, you could argue in some respects, like a mini little Fiddler that runs inside of your process or inside your web application, and it's able to service web requests; and so you can create applications that work perfectly offline using ServiceWorkers.

But ServiceWorker, because it has the ability to intercept subsequent network requests and respond with whatever it likes, the ServiceWorker itself obviously needs to be delivered securely. In order to help achieve the vision of having a web platform that's fully powerful and fully competitive with native applications, we need sites to start moving to HTTPS in order to unlock features like geolocation, ServiceWorker, and the like.

Beyond that, we're also trying to raise awareness of the dangers of non-secure applications in general.

People don't really think that attacks are going to happen to them.

They think, "Hey, the attacks that you're hearing about, the revelations from the Edward Snowdens of the world and the like, are only against people that are doing something super interesting."

But the reality of the situation is, bad guys will make a buck any way that they can; and if they can do an attack that injects JavaScript into webpages that's going to trigger some funky behavior in your browser that tricks you into running malware, or clicking on their ads, they'll do that, and we're starting to see cases.

There was a really great paper in the spring by some researchers that found that there are some interesting attacks that can be performed at the TCP/IP level, where they can control some of the content that's going over HTTP links that aren't necessarily a man in the middle, they're more like a man on the side. They can serve malicious JavaScript from pages they don't own; these sorts of attacks are obviously devastating.

Guy: Yeah, and I think, basically that whole sort of continuum and evolution that you've described is interesting. I'd like to sort of pick it apart, because there are a whole bunch of interesting sort of entities here, right?

At the beginning, you were talking about how the complexity of the applications and the value of them has evolved, making them a more rich target, right? More opportunities to get in, and more value if you did. I like that, I like the analogy, right? The reality is that we have, it's true today that you can argue around what's the situation in mobile; but on the web, we definitely increasingly use very few, or a very select number of desktop applications, and it's all web applications.

And I like the cyclical nature, I don't know if 'like' is the right word, but I agree with the analogy of the cyclical nature of some of

The security controls that we have on operating systems need to move to this browser operating system

and needing all of the vetting and verification.

I definitely feel like the same evolution happened on the operating systems themselves. It started off from securing the operating system, and it continued on to help protect the applications on the operating system from being hacked, or being a mechanism to spread viruses; and now the web world is undergoing a similar transition, where the browser is the operating system. It's not just in charge of securing itself, but also securing the applications on it.

I guess the good news is that the browser is actually in a much more controlling position; and to an extent, I love ServiceWorker, I love these new capabilities that are coming into the browser. I think I am pretty firmly on the web side of the web-versus-native, in terms of where I think the evolution and the path is going; and through that,

I believe ServiceWorker and some related technologies are amazing ways to bring native-like experience and more into the world of the web, but I am afraid of it from a security perspective by a pretty large degree.

I haven't opened up HTTPS quite yet, right? You sort of touched on it, and I think that one is worthy of a lot of conversation by itself. But in general, it feels like there's some contention between the new functionality that we're adding and how it's so powerful, and the sort of security risks, I guess, that we're exposing in the process.

I'm curious, when you see the work going on inside the Chrome security team, as well as when you talk about giving advice to web developers building and trying to use those technologies; do you have some insights around how to judge those trade offs? I mean, how to choose whether you want to tap into some cool new feature, a new capability, and how to understand the security risks and does it happen that you have great new functionality ideas in the browser that you end up disqualifying for security reasons?

Eric: Yeah. Certainly, it's always been a hard trade off; and I like to think, and I hope, that the trade off is harder for designers of web browsers than it is for site operators building sites; because unless you're trying super-experimental stuff behind flags, in general, if a browser has an API available, we want developers to be able to use it securely and not shoot themselves in the foot with it.

There's two aspects of that. When I worked in IE in 2004 and so on, we were extremely hesitant to add platform APIs to the browser because of the fear that, "Hey, we can't make this secure, we can't ship fast enough if there is a problem, to fix that."

So we were extremely conservative around adding APIs, and certainly, there were times when platform feature teams said, "Hey, we want to add X to the browser," and as a security team, we pushed back pretty hard because we were very concerned about the security implications of that.

The problem is that's not really very sustainable.

You can't neuter the platform and hope that the platform is still going to thrive and live on.

You have to be very thoughtful about the principles under which you're operating, the trade offs and balances, and find ways as much as possible that you can deliver new features in a secure manner.

Sometimes, the relative change that you need in a feature in order to make it less interesting to be abused is fairly small, whether it's a confirmation with the user, which sometimes there's user fatigue, or feeding things through updated services, where you can keep track of legitimate uses of APIs and things like that.

We've got this feature now that we're starting to allow origins to opt into trials of new features and things like that, so that we can get some real-world experience in using them before we're unlocking them for the web as a whole. I think that that's kind of the trick.

We definitely are giving users more of an opportunity, or developers, as it is, developer-users of our APIs, more of an opportunity to shoot themselves in the foot; and so one of the best examples of this probably is what happened with cross-origin requests.

Same-origin policy in browsers is really all about isolating sites from each other, and not allowing interactions; and the problem, of course, is sites want to interact with each other. You want to have your APIs calling APIs from other sites, and so same-origin policy becomes a real problem.

And as we looked at this in the 2006-2007 era in IE, we felt that the model that had been proposed for cross-origin, which was the dominant model in Flash: you've got this cross-domain XML file. In the cross-domain XML file you have, effectively, what resources should be accessible cross-origin.

There were some notable mistakes in configuration on the part of important sites: I think with Flickr, briefly had a cross-origin policy that said effectively, "Any site is allowed to get any data it wants from us." You could, if a user browsed to a malicious site, that site could go out, crawl Flickr, and take all that user's photos.

In the IE team, we were very hesitant to do anything like that, have a system whereby a user could shoot themselves in the foot in a wild-carded way; so we created this object called XDomainRequest. The notion behind XDomainRequest was, it was sort of a nerfed mechanism for getting data cross-domain.

The security principle that we used in building cross-domain request was effectively syntactic sugar: we didn't want to have anything that could go to the server that the browser couldn't have sent some other way.

You were limited to GET and POST, you couldn't use other methods, you were limited in the types of content types that you could send to the server, and things like that.

The notion was, "Hey, if a site is vulnerable to XDomainRequest, they were probably vulnerable in some other way." For those with an understanding of web browser history, they'll know that XDomainRequest has been relegated to the dustbin of history, because no other browser adopted it.

Instead, a proposal called CORS, Cross Origin Resource Sharing, took off instead. CORS was effectively the Flash model, whereby a website declares via policy, the policy is delivered slightly differently; but they declare it via policy who's allowed to access them.

The reasoning is that cross-domain request was not powerful enough for some of the scenarios the web developers wanted; and web developers will say, "No no, we'll be real careful, we'll do it right." And, once other browsers adopted that object, it was over for IE, and IE has now adopted CORS as well.

It's certainly true that you can still shoot yourself in the foot with CORS. A security researcher from I think Cloudflare recently wrote a paper finding sites where the sites had said, effectively, access control allow origin, and they would reflect back whoever was talking to them, and they would allow credentials.

Effectively, they'd configure their policies such that they had the same problem that Flickr had. Anybody could ask for any data they wanted, and the server would return it to them. There was a site, some sort of ordering site, that he used in his demo, where if the user was logged in to this site, the malicious site could reach out and grab the user's profile page, and grab the user's home address, phone number, credit card digits, and so forth, from this site, because it had been misconfigured.

So certainly, there are opportunities that have been unlocked as we add platform APIs where developers who aren't careful can shoot themselves in the foot. But for the most part, browser designers are working pretty hard to make that as difficult as possible.

We definitely want to add powerful capabilities to the platform: we don't want to have the Fisher-Price platform, with no sharp edges, because it turns out sharp edges are useful for things.

But we want people to know when they're using the sharp edges. One of the things that's been sort of fun is there's a feature, I like to call it sort of DEP/NX for the web, called Content Security Policy, where you can throw away permissions and say, "Hey, I don't want script execution coming from inline-blocks on my page."

That feature has a couple of directives that negate the security benefit in one way or another, and those tokens are prefixed by 'unsafe.' If a site wants to use a Content Security Policy, but they want to do something that's not secure, they are literally typing 'unsafe' into their policy. Hopefully, a developer is going to say, "I don't really understand why that's unsafe," and they'll go look further to understand what's going on.

Guy: I think a lot of it is about secure defaults and making it very explicit. I gave a talk with Rachel Ilan Simpson, who was a designer on the Chrome team in the Munich office. We gave a talk titled Security Ergonomics that talks about, all about how users make insecure decisions.

As we analyzed a bunch of use cases and a bunch of scenarios, a lot of it came down to what do you expect the user to do with information that they have? What type of knowledge or insights do you expect them to have ahead of time? I think it's entirely legit to say, "I expect the user to have sufficient reading understanding, that they know that if they write 'unsafe' in the name, then that thing that they're adding now is probably not a safe thing."

On the flip side, understanding the security implication of writing even an asterisk is some course field is just not quite as easy, right? You need to think more broadly, you need to have a better understanding of the implications. We could probably talk for hours and hours around all the security controls that browsers have.

I'd like to switch over to HTTPS, but before I do that, what's a good place that you'd recommend for developers who want to take advantage of all these different security controls, from the Cookie Prefix, to the Content Security Policy, to a bunch of other security headers today. Do you have a recommended location, a recommended web destination, or something like that, where these are well-explained and inventoried?

Eric: There's no shortage of information on the internet, for sure. The Chrome team actually puts up a fairly substantial amount of documentation for web developers: not just related to security, but also for performance and things like that. The one thing I would encourage in some cases, certainly for developers that expect they're going to make a fundamental change in their security, is to have a look at the spec itself.

It feels kind of super 133t and nerdy to actually go read the specification, but sometimes you'll find things in there that are really interesting and sort of motivate, explain the motivations behind the design that sometimes gets lost when you read the BuzzFeed article of "Five Things To Put On Your Site Today!" The specs can be good.

In terms of practical guidance, there's definitely books that have been written about web security. There are some great ones just about the fundamentals, so things like The Tangled Web, which kind of explain the security model of the web, and some of the places that had shortcomings.

Many of the features that have been designed since that book was written, are designed to combat some of the problems that are mentioned there. There's also some scanners: they're interesting. You've got things like securityheaders.io, and Mozilla's Observatory, which came out toward the end of the summer.

You give it a URL, and it will go out and look and see what's in use on that URL in terms of security features. It'll say, "Hey, you're not using Strict Transport Security, you're not using X-Frame-Options" and so forth.

My hope for those tools is that they continue to evolve beyond scanners, but also to help the user understand the importance of the security features; and so saying, "Hey, you're not using X-Frame-Options, so we're going to deduct points from your score" is not necessarily as useful as explaining, effectively, "X-Frame-Options is about UI redress attacks. Users might be clicking in your site, thinking they're clicking on somewhere else."

Providing that mapping is something, I think, that these sites will continue to evolve and help users understand the importance of the feature that's been cited as missing.

Guy: Cool. I think those websites are super useful, and I guess we can throw out a link to them as well in the transcript of the podcast, where people can check them out. I agree with you around understanding the concept behind them: there's a little bit of a trade off, right? To an extent, you want to make them secure, you would rather they be secure blindly than they be insecure blindly.

You would rather users be secure with eyes wide open.

Eric: Yeah, absolutely. One of the challenges we have, and in particular, as I work with sites in HTTPS migrations, is its really important that sites don't break.

This has been true from the beginning of time: desktop applications had the same problems. As we introduced features like Data Execution Prevention, and noexecute in Windows, and ASLR, and things like that, we couldn't just blindly turn them on for everybody, because applications would break, and they'd say, "Well, just don't install the new version of Windows," or "Don't install XP SP3," which is a disaster for security everywhere else, but it breaks our application.

In the web platform, we have a sort of an obligation in some respects to be insecure by default, just to make sure that historical sites, sites that haven't been updated, continue to work. We need sites to opt in to these directives in order to get the protection, but we also need to have them test their applications.

With HTTPS in particular, and maybe perhaps as a segue, we've got two features: HTTP Strict Transport Security, and HTTP Public Key Pinning, that are very powerful in helping sites protect themselves against important attacks. But they're also foot-guns.

Effectively, you can create a denial of service for yourself, where you send one of these directives with a long lifetime, and say, "Hey, for the next year, don't allow anyone to connect to my site except over HTTPS," and "all the security scanners tell me includeSubdomains is a good thing, and I don't really know what that means, but I'm going to do that too,"

And then they turn on that policy, and oops! They forgot that they also have admin.example.com, and admin.example.com, for whatever reason, isn't using HTTPS; and I can't get to the login page for admin anymore, and my site's broken, and it's broken for a year, or until I manually tell all my users to go clear out their configuration and things like that.

We've been doing more to try and help developers avoid the foot-guns, and so things like HSTS and HPKP, are designed to, the notion is, you're going to start it out with a very short lifetime, and so when Google turned on Strict Transport Security for Google.com, on the main site.

I think when they first turned it on, I think we used a five-minute lifetime, or a ten-minute lifetime, and watched to see what would break and what would continue to work. Those experiments actually, very often, turn up sites that you'd long forgotten about; and so one of the pieces of trivia was, I think we broke the Santa Tracker the first time that we turned on Strict Transport Security, because Santa Tracker was not a secure application; and oops, now kids can't look at Santa.

So, you know, that got fixed relatively quickly. But certainly, things that are more critical to your business, you want to make sure those things aren't breaking.

There's opportunities to use security features incorrectly, and hurt your site. We definitely don't want that.

Guy: Yeah, of course. You want to walk that line, I guess, to know that many of the security features actually have a little bit of the advantage in this extremely over-backward-compatible web world. Some of these security mechanisms actually have the advantage of not requiring full adoption, right?

You can use them with these headers, like HSTS, despite the fact that it is not supported by all browsers, because that's fine, it's a security mechanism that would reduce your exposure for those browsers that support it; but at the same time, you have to be wary of the ones that are still quite broad in implication.

You opened this up-- It's probably going to be our last topic here, is HTTPS, a topic that's near and dear to both of our hearts. I've recently done a study that talks about HTTPS adoption earlier this year, and as a whole, it was a positive. It talked about how HTTPS adoption has done in one year what it took 19 years to do prior to that, in terms of adoption.

At the same time, if you look at the absolute numbers, you still see that a very, very small percent of the web, we're talking about under 20: depending on the stat you're looking at, probably under 13 or 12 percent of the web, is using HTTPS. So the trend is positive, but it seems to be advancing quite slowly.

I think the move to HTTPS has a lot of pros and cons to it. I was wondering, in your view, you deal with this more than most people on the planet. What would you say are the key motivators to moving to it, and what would you say are the easiest first steps? How do you advise a company that is sold and wants to move to HTTPS, to make it so?

E - In an ideal situation, the key motivators are the reason for HTTPS to begin with. Confidentiality of the information that's being sent and received, so that people on the network don't know what your users are looking at, don't know what they're reading. Integrity is a very important one of those.

You want to make sure that the application, the website, the things that your designers have worked really hard to deliver is the one that the client actually sees; and one of the things

I've started calling HTTPS 'hi-fi for the web.'

You want a high-fidelity experience where the user's actually seeing what your designers wanted. We've seen cases where you're on a captive network, where ads are getting injected by the network itself into pages, and so there are some screenshots. You're on apple.com, and there's an ad for something from Best Buy.

Obviously, Apple didn't put an ad for Best Buy on their website, but it has been injected by an ad network that is man-in-the-middling traffic and rewriting non-secure traffic with these ads. You want these protections for your users.

The challenge we have is that some sites are not necessarily focused on their users' experience to the degree that we in the browser world would like; and so carrots, effectively, have been added. Powerful features are starting to be added behind checks for HTTPS, so geolocation now is a sensitive operation: we don't want websites to be requesting this data if they haven't been delivered securely, and so we require HTTPS.

There's also, now, a performance element of it. In particular, two of my favorite features in the browser, HTTP/2 and Brotli compression, both require HTTPS; and the reasoning for that is not really the strict "Hey, we need security for this information." It's about, "Hey, we need integrity for this information."

The reasoning is that there's a lot of gateways and proxies out there that will go do really weird things to your network traffic, and those things were written in an era where HTTP/1.0 and 1.1 were dominant; and if those things touch new traffic, if they touch HTTP/2 traffic or Brotli traffic, they end up corrupting it.

Google saw this, and other browsers saw this first with WebSockets. WebSockets are TCP/IP sockets that are run over sort of an HTTP handshake, and then you have a bi-directional stream; and it was found that many gateways would actually look into their traffic on the socket, and if it looked like HTTP traffic, they would start doing really strange things, like caching bytes from the middle of the WebSocket.

So they created this sort of weird masking system for WebSockets, where the data is obfuscated so that the gateway doesn't see it and manipulate it. But really, a better approach is to just use HTTPS to begin with. Similarly, for HTTP/2, the faster, next-generation transport for the web, those streams just are not going to work properly if the gateway is messing with them at all.

While technically, the specification does not require TLS in order to use HTTP/2, all the browser implementations do require secure transport, so that that traffic is not manipulated. And we saw this, those problems of manipulation as well, with compression. Google, at this point, probably many years ago, I think, it was eight years ago, came out with a new compression algorithm for the web called Shared Dictionary Compression for HTTP.

They found that there were gateways and proxies that would, if they saw SDCH, they would say, "Oh, they spelled G-Zip really funny, so I'm going to try and G-Zip decompress this, and then remove the content encoding header;" so clients were getting corrupted content that they couldn't decompress.

That's why Brotli, which is a new compression that gives significantly better results than G-Zip and DEFLATE, Brotli requires HTTPS for the browser. The browser will not advertise its support for Brotli over a non-secure connection. We're making faster sites by having high-fidelity as well, and that's an important thing.

Those are both pretty motivational. One of the other things that we're hoping is that sites are starting to understand, "Oh, well, my users are actually concerned about privacy," and so you've got sites like BuzzFeed. BuzzFeed is not your bank, they're not your retirement company; but BuzzFeed has readers, and readers want to be able to read articles about topics that may be sensitive, particularly in their region.

There's places in the world where reading certain articles on BuzzFeed could get you in trouble with your employer, your government; and they want users to be able to read whatever they want, and so BuzzFeed deploys HTTPS to help ensure that their users are not getting spied upon.

Those are all the incentives that are being exposed. But on the other side of the equation, we're also trying to reduce the roadblocks; and a huge one of those is the advent of Let's Encrypt. Let's Encrypt is a free certificate authority that delivers certificates in an automated fashion to servers; and this has unblocked a huge number of sites from moving to HTTPS for their customers, sites like WordPress.

You can get a WordPress account, and it's a check box to turn on HTTPS for your server. Same thing for DreamHost. It's not just those mass hosters with hundreds of thousands of domains: it's also smaller hosters are going out and saying, "Hey, we can turn on HTTPS. It's not going to cost us anything."

The marginal complexity is low, particularly if you're using one of the platforms for which plug-ins that automatically handle Let's Encrypt and turn it into a check box feature are there. And we're excited about the number of Let's Encrypt certificates: I think I saw, recently, there's like five million certificates that have been issued.

That's going to make a huge difference. Now, to your overall point,

We want everybody to be on HTTPS; and the web is enormous, and it's going to be a long time until we get there.

We definitely want sites to be thinking about making that move. We want to help them make that move. One of the exciting factors is, not the raw number of sites, but the percentage of the time the user spends in their browser for which they are on a secure transport.

With major sites like Twitter, and Facebook, and Google using HTTPS for everything, this is starting to mean that most users are going to spend most of their time in the browser on secure connections.

It's going to be a long time until we clean up the long tail of the archive of the Geocities website from 1996 that's out there and not HTTPS. Certainly, if most users most of the time are in a secure position, I think the world, as a whole, is going to be better off.

We may be in a position to make some of the attacks that have plagued people over time essentially not lucrative and not financially rewarding for the people that would perform those attacks.

If all the user's sensitive operations are happening on an HTTPS connection, passive surveillance of that user becomes far less interesting, because I can't tell anything meaningful about them. I may know, "Oh, he seems to be interested in Battlestar Galactica fanfiction written in the 90's;" but I'm not going to know how much he's got in his bank account, how much his profile is set up on a dating site, things like that.

It's a long road, I definitely took the HTTPS evangelism job, feeling a certain level of job security. If we were at a 100% HTTPS by the end of the year, I'd be a little nervous. But that would also be a great place to be in.

Guy: I'm pretty sure Google would find some interesting things for you to work on in the very unlikely scenario.

I think that's also sort of a nice loop closing here from one of the points you made earlier on, about the fact that everybody's constantly under attack. And it's true that if you have some small blog or if you're not someone who is very prominent, you might not be a dedicated target, but you are an opportunistic target to tools that are around.

HTTPS is one of the best tools as a consumer and as a website to just reduce the attack surface, reduce the opportunity for someone to either steal your data, or sometimes compromise the information that is received to your machine. Hopefully, we see really good adoption. I'm super excited about it: I also have kind of opposed and am rating some of these things, the carrots, the sticks.

I think somewhere in there is also the fact that Google, from a business perspective, that Google has kind of announced a few years back, a couple of years ago or so, that it is factoring in HTTPS into its security algorithms. Very few things move the needle in terms of web adoption of a technology, or some component of the website than the SEO, the Google ranking of it. I think there's definitely a lot of goodness there.

Hopefully we see a lot of really good growth and increase, see that curve go up and to the right in terms of the percentage of websites; and you're right, maybe the more important stat, which is the percentage of time that users spend in HTTPS versus HTTP.

Eric: Yeah, you need both. We definitely want people to be on secure sites as much as possible, but eventually, we need them all; because any non-secure navigations you're performing, if there's an active bad guy in the network, he can intercept those and do really bad things to you.

We certainly want them both. In terms of the up and to the right, beyond the blog post that you wrote, I think it was over the summer, around HTTPS adoption. Google maintains an ongoing updated HTTPS transparency report, where we show the percentage of our traffic that's been done over HTTPS, and that's gotten a great trend, particularly with major sites like YouTube adding a huge amount of traffic to that.

We also look at the top 100 sites worldwide and track their adoption of HTTPS. Since we launched that report, I think it was in the spring, the numbers have been really quite good in terms of improvement. We're seeing more and more of that over time, and certainly we're doing our part to help evangelize that with things like our Progressive Web Applications outreach work, to try to get sites to build sites on Progressive Web Applications and ServiceWorker and the like.

Guy: Cool. Yeah, we'll definitely post that link up on the website as well. I'm definitely sort of interested. I think I somehow missed that one, so I definitely want to see it. Well, I think we're over time here already, for what we intended. But this was great information, and great insights here.

Thanks a lot, Eric, For joining us here.

Eric: Thanks for the opportunity.

Guy: Keep up the push for HTTPS and security and I'm sure we'll share many circles in that journey.