Ep. #4, The Long Road to Video.js 6.0

Matt McClure: Hey everybody, welcome to Demuxed. This is our first podcast of 2017. So sorry for the hiatus there, but it took us a solid four months to recuperate from Demuxed, the conference last year, which went well, thanks for asking. So before we jump into the larger discussion around the excitement on Video.js 6.0 and all that sort of stuff, wanted to talk a little bit about Demuxed 2017, which is already officially happening.

We are planning so far ahead you guys. So it's going to be October 5th at Broadway Studios in San Francisco again, so if you're available you should come. So we haven't opened up the call for speakers yet or anything like that, but expect that like mid spring probably.

David LaPalomento: So wait, Matt, is Broadway Studios much bigger than the Crunchyroll space that we had for last year?

Matt: Yeah, we're aiming for like 250 people or so, so it is a little bit bigger. We were kind of packed at the gills last year. So this gives us a little bit more breathing room. Now there's like actually places to hang out that aren't in the main speaker area if you want to.

David: There's enough space to like hold your food while you're eating it?

Matt: Yes, yes, there's enough space to actually go walk through a lunch line and everything.

Phil Cluff: Hang on, you didn't enjoy that last year? That was the highlight for me.

David: You get to rub elbows with people, literally, so yeah, it was great. Demuxed is too popular, that's the problem. It's a big deal.

Matt: And we're a bunch of slobs. So this year, again, October 5th, put it on your calendar, plan to come out. So today we've got two of the core contributors from the Video.js project. If you're not familiar, it's an open source HTML5 player that's a fairly popular project on the interwebs used by a bunch of big companies. And if you're not, I think we've talked about this in the past, but if you're not familiar Steve started this thing probably what, six years ago, Steve?

Steve Heffernan: Almost seven, yeah.

Matt: Back when he was left alone in a house in the mountains when they were working on Zencoder. So, lonely.

David: I was checking the commit log, and it looks like Brandon Arbini actually gets the first commit. So I don't know if you get that. Yeah, I know. He was in there with like create repo.

Matt: I know that irks me every time I see it now, but at the time I did not know how to use GitHub. So every Video.js user out there, you heard it here first, you're in good hands. So yeah, Steve started this thing off back when HTML5 video was obviously a terrible idea to try to use in production, and then I started helping him on the project when I was at Brightcove as well.

And then we started working with Gary and David, who have kind of taken the mantle of really like the larger project direction these days and done an awesome job of kind of shepherding things. So why don't you guys tell a little bit about it yourself?

David: You want to go Gary?

Gary Katsevman: Sure, I'm Gary. I started on Video.js mostly just entering issues first. I was kind of avoiding writing code on the project itself initially. And also when I was questions I was mostly like, I know this part for sure, so I can answer it, and as I was answering more questions I started looking more into questions that I wasn't sure of the answers to and investigating them and then answering, and so I grew my knowledge and I was more comfortable making changes to the Video.js, and now Im making huge sweeping changes to the code base.

David: So I think my introduction to the project is actually kind of similar to how a lot of other people come to it, which is at Brightcove. We had a Flash based video player that was very successful, but we knew that that wasn't the future, and we were thinking about replacing it, and we said, "Well, should we rewrite it from scratch or should we look at one of the open source projects that are out there?"

And Steve at that point in time had started working at Brightcove too and it seemed like a really natural thing to do is use the open source video player that's already hugely successful and base our efforts on that. So that's kind of how I got involved in Video.js.

Matt: That second voice was David LaPalomento by the way.

David: Oh yes, sorry, I should've introduced myself, hi.

Matt: So Phil, have you actually committed code, or have you just written sash.js and stuff like that?

Phil: Sash.js and raised a bunch of issues.

Matt: It's basically committing.

Phil: I'm still on Gary's side, right, raise issues.

Matt: Yeah. So I expect everybody that's listening to this at least knows what HTML5 video is, but I guess it makes sense I think for Steve to give us a little bit of insight into what the world of online video was like when Brandon committed to the project.

Steve: Well yeah, let's see. So I started building the project as mentioned back in 2010. We were going through the Y Combinator program, so the startup accelerator, a company Zencoder. We rented this house in the middle of the woods, you know, in the hills behind Los Gatos, because what you do with Y Combinator is you go move close to Mountain View so you can go through the program. So we rented this house for three or four months, and my co-founders who both had families in the area would go home on the weekends and leave me in the house in the middle of the woods.

Phil: It's the start of a really bad horror film, actually.

Steve: Yeah, wander around the house by myself. So one weekend we had been talking about all of our customers at the time for Zencoder were using Flash Players, and people had started to just barely start hacking around with HTML5 video. I mean this was beginning of 2010 and browsers started releasing HTML5 video like late 2009, so this was super early, like 20% of users actually supported HTML5 video.

I really wanted to build a video player with HTML and CSS as opposed to hacking around in Flash.

David: Everybody did.

Steve: Yeah, I mean it was like a really amazing idea. We're doing all this web technology stuff, and then once you jump into Flash you have to learn this like completely other language and wave building thing. So it was a really cool idea to be able to hack around with that. So I took one of these lonely weekends and I just started building controls for the HTML5 video player. And it was one of those projects where the time just kind of flies by, because you're just drilling through it and days just kind of bleed into each other. I remember it like vividly. It was an awesome weekend.

Out of that I built all the controls and made like a tutorial, like I released a tutorial on my blog. I was like, "here's how you build controls for HTML5 video". And then that got some traction and then decided to kind of spin it into a library that somebody could just drop into a page. I remember when that first got its big piece of publicity. So I'd built a website for it, do you guys remember Daring Fireball?

David: Yeah.

Phil: That's so great.

Gary: It's still a big thing.

Steve: Most of the blog now is just hating on Apple haters I think.

David: Oh yeah, yeah, yeah. It's an uphill battle for him, yeah.

Steve: So yeah, so Daring Fireball just had a three line blog post that was, "Hey, check out this new HTML5 video player. It already supports WebM". This was the tagline, which was kind of funny because like it supported WebM because browsers supported WebM. It was like I didn't have to do anything, but he was like, "Oh my gosh, this thing already supports WebM!" It was like, "Yeah okay, well I'll take it."

David: I guess at that point in time he wasn't using Safari to browse the Internet.

Steve: No yeah, I would guess not.

David: There is no way that worked.

Steve: So yeah, I remember that weekend after he talked about it the website got, I think we got like 20,000 visitors that weekend, which like totally blew my mind, right? I started envisioning like, "Oh, that's like half a stadium's worth of people, and like trying to envision what that would look like, and I was like, oh my gosh, people are actually like using this, or at least like it enough to go check it out".

Matt: At this point he's alone in the house pretending he's in front of this audience. "Ah, Video.js ah!"

Steve: Which is funny, because like today we get like 20,000 visitors to the website every single day. But yeah, so that gave us a lot of motivation to put more effort behind it and just kind of keep building it out. Yeah, I think maybe a year later we built kind of a Flash shim for it. Like up until that point. So when I first built it you could like fall back to Flowplayer I think was the player that we used.

So if it supported HTML5 video it would use that, and then otherwise it would just fully fall over to Flowplayer. And then like a year later, or maybe it was a couple years later, we built like a Flash shim that was just like this super lightweight, we still use it today, super lightweight Flash player that just kind of mimics the video element, no controls in it or anything. We just overlay the HTML controls on top of that.

David: Steve, do you know is anybody else doing that still? Like I know we are now actually like heading towards retirement on that little piece of Flash, but I feel like I don't know if any other video player even now adopted that strategy, and I always thought that was one of the coolest things about Video.js.

Steve: I think every player has now, honestly.

David: Are they doing that?

Steve: Yeah.

David: Oh okay. I really felt like most of the alternatives, well, I guess you're right.

Steve: It was relatively recently, if I'm getting this right, I could be totally wrong, but I think Flowplayer completely rebuilt their thing, and I think it's built the same way. I know JW Player, JW Player at least now supports like CSS styling, which I assume means that they're doing this, but I can't say for sure.

David: Well it took a while, let's put it that way.

Matt: For listeners that aren't familiar.

We're talking about about the control bit here. When Flash players ruled the world, trying to modify and style them was difficult to say the least.

There were things like ActionScript. You were either directly in ActionScript.

Gary: Load custom .swf's.

Matt: Custom .swf's, or like if you were lucky you were using something like Brightcove.

David: BEML.

Gary: If you want to say that's lucky.

David: It was better than writing ActionScript.

Matt: Building a player that looked the way you wanted it for your website was impossible for the layman, and like a giant pain in the ass for even really technical people.

So the big innovation here is that Steve hid the controls and then built custom ones on top of the player using CSS. JavaScript events were the controls now. Flash player didn't have controls.

It just sat under these HTML and CSS, HTML elements styled with CSS, which sounds like a no brainer in 2017, but I remember trying to style Flash players.

Gary: And the HTML Fullscreen API made it so that you could actually take the player full screen and actually have controls. Because previously you could only full screen with Flash, and if you weren't doing the controls in Flash in full screen you lost controls.

Steve: There definitely was a lot of downsides. Maybe not a lot, but there were some significant downsides to that approach. Like you couldn't go to full screen originally. You could do like full window, but you couldn't do full screen. There was also like some performance issues of like overlaying HTML on top of the Flash player, especially when you talk about, I forget what it's called, but there's some like mode in Flash when you're playing video.

David: Wmode. A lot of the heartache internally was over using I think it's, there's opaque, transparent, and whatever, and you couldn't get hardware decoding unless you were not using the variant of Flash which let you overlay HTML on top of it.

Steve: There was good reasons why people would decide not to do this, but I was just incredibly bullish on HTML5 video. Like I wanted it to be the future, so I was just going to like make it today and then figure out how to deal with the problems after that.

David: Yeah, well I guess it worked.

Steve: It came around. Back then it was like 20% of users actually supported it. I would go and talk at like Streaming Media East and West, those conferences, and I would get pushback on like, HTML5 is never going to be the thing. Why do we need this? Flash already does everything we need it to. It's fine. Stop messing with our pipeline. There was a lot of pushback on HTML5 video in the day.

David: Those people might still feel that way, but yeah.

Steve: Yeah, absolutely.

Matt: When did you start at the BBC, 2012, 2011?

Phil: Somewhere around there. Yeah, BBC was a Flash shop for a very long time. The BBC only switched to an HTML5 default player less than six months ago, I think, for desktop, because they never did a HLS polyfill. So they went kind of from, they went via HDS, or they went via HDS, so had kind of that little bit of extra Flash buy in. That one that I think we all kind of bypassed it a little bit, real world.

David: Hopefully. I think it was interesting because the player was one thing, and getting it to look good in your webpage.

But that was like a period where there was all this infrastructure around serving Flash video. And the HTML world didn't have adaptive streaming, and you hugely popular deployments of massive FMS server infrastructure, and people didn't want to give that up.

Phil: And the CDNs had spent years building out these massive RTMP streaming farms, right, and then suddenly no one wants that anymore, right?

David: And it was awesome lock in for them, right? So instead you were switching to a world where like you didn't need to have this incredibly proprietary CDN. You can have an HTTP server and just serve up files. And like actually I would say like the emergence of like Fastly, and I think CloudFront actually does have FMS service, but a lot of these like new CDNs I think are a result of that change.

Phil: They just want to be a HTTP chunk server, right?

David: Right, exactly. I think it's worked out for everybody, but it was like really painful the transition.

Matt: I would say the project's goal in 2010 was to like show people that there was a potential other cool option. It was a fun hack, honestly. Then like 2012, 2013, it's like a viable option for people to assume that they're going to be playing HTML5 everywhere.

Steve: At that point people had to make a choice, because you had to use HTML5 video on mobile. So it's like you either like falling back to HTML5 or you're falling back to Flash, one of those two options.

But the project's goal was still ultimately to give you an easy way to build custom players that are styled the way you want and will work everywhere

Because eventually it'll ultimately fall back to Flash with a common API that works across browsers. Which is still obviously the goal, but like it feels like it's shifted in the modern world where the Flash fallback, as we'll talk about later, is becoming less important.

So I'd be curious to hear from you two, what would you say is like, what is the project's goal in 2017? If you were talking to people on the issue tracker today, what would you say is like the most important need for Video.js now?

Gary: I think probably the biggest thing for Video.js is to have a consistent API across all browsers with a consistent look that also allows for you to easily plug in into the player so that you can support multiple other formats and have extra functionality that you can write and have it so that when you write plugins you don't need to, okay, writing this plugin, let me test it in Chrome and do Chrome specific stuff, and then when I'm in Safari do Safari specific stuff. You can just use the Video.js API and it will be available across platforms.

David: I definitely agree with that. I think also the challenges have changed over time, so yeah, it's not like looking like a video element where Flash is anymore. It is about doing more interesting things and supporting more interesting video.

The emergence of media source extensions and the ability to do adaptive streaming formats in HTML is another area where I think Video.js is trying to offer the right set of primitives to make that possible.

And put it together in a way that you don't necessarily have to know how to write adaptive bit rate switching algorithm to play a video.

Phil: I think that's really interesting, actually, isn't it? Because we kind of consider what the hard bits are right now and one of kind of visions of like the video elements. If you look at video element from an Apple perspective, I've got a video element. I can trigger a HLS manifest URL into that video element, and I get adaptive streaming, I get resizing, I get all these toys and bells and whistles. I think when we look at Video.js we have to think, well okay, let's make that experience but with everything, consistent across all browsers and everything.

David: Yeah no, I think that's a really great summary of like where are the challenges moving with the video element.

Matt: I'd even say to add to that, the last episode of the podcast we had Owen, AKA Snowen on GitHub, come in and talk to us about accessibility and the challenges around accessibility, why we should be doing it, and so on and so forth around accessibility last episode, but I think that's also a big deal for modern Video.js is like that's been the biggest leaps and bounds that I think I've seen in the project in the last year or so.

It feels like have been around these massive improvements in terms of accessibility, like the caption settings box thingy, the fact that captions actually work on iOS now. You know, things like that are, I feel, like a big gain.

David: So that's so funny, because that's one of those things which is supposedly baked in, but like I feel like we still regularly get a bunch of questions about, Hey, how come we're not using native text tracks on this platform versus that platform? In Video.js we like emulate them through JavaScript a lot of the time. And it's really funny, but as soon as you like scratch the surface on those accessibility features, they're really busted all over the place. So it would be great to use native stuff, but I guess I'm trying to think of a good example of like problems.

Gary: Well for example, for a long time we couldn't use native text tracks in Firefox. Because the way that Video.js worked is it takes the video element and then wraps it in a div which is where we put the controls, but to do that we actually remove the video element from the DOM and then put it into that new div that we create.

But Firefox had a bug that like v6 now in Firefox 50 I think, where text tracks that were loaded in the DOM originally were canceled from loading when the video elements removed but then never told to reload when the video element was put back in, and so writing workarounds to support native text tracks in Firefox for that would be smithy more work than just saying well we're just not going to use native text tracks until Firefox fixes this bug.

Steve: Yeah, and it also had the nice advantage of like you can sort of style the emulated text tracks, which I think is something that some people wanted for compliance reasons, but other people would just like their font to not be just pure white on pure black for their captions, and there's like nascent CSS selectors to do it, but they don't work all across the board, and they don't always work very well, and so, you know.

Gary: Or they don't work like you expect them to work.

Steve: Yeah, yeah. Trying to write CSS that works across browser that does what you want for text tracks is like unbelievably difficult.

Gary: And we actually found bugs relating to that. For example, I remember seeing on Safari if you were to select the queue there's the pseudo of selector ::q, which is supposed to select the queue, but if you add a margin bottom to that because you want to lift the captions up, Safari is really weird where it applies it like each frame that it loads the player.

So basically each time that new captions are refreshed, the caption would go down and then back up. So basically you get unapplied captions written out, and then they get applied and they'll move up. And then sometimes it gets applied twice, so it ends up being like, say it was down one em, it'll end up being two em from the bottom. It's bizarre.

Steve: This is the last caption bug we're talking about. So we were testing 608 captions in Edge because Edge has native support for HLS and it supposedly has 608 support. 608 captions, if you're not familiar with them, are like the captions that are actually embedded inside of the video files. They actually kind of like are interspersed among frame data in video files.

It turns out that Edge does, in a sense, support 608 captions, but what it does is 608 captions have a frame number on them, it will display them for that frame and that frame only, and then get rid of them. So every caption shows up for 1/24 of a second, which isn't enough time to read them just in case you were curious. I'm sorry Edge people, I'm sure that that was not a fully baked feature and it just got out of the wild, but it was crazy.

Gary: It's captions for speed readers.

David: Yeah, yeah, if you're really fast they're great.

Gary: Tying it back to Video.js, this is the thing that we spent so much time on is finding these bugs and then figuring out what can we workaround them so that users of Video.js don't need to deal with this ever again, and I think for text tracks we have a pretty good solution at this point.

Matt: Yeah, I think if I had fully understood the complexity we were getting into, I would've never started the project.

Gary: You said building a video player was easy.

Matt: So I think as a user I definitely want to fix all those text track bugs myself, so I would never use a project, but I can see why other people would want to. Yeah, so as we move into 2017 we're talking about a new era in Video.js now. We're moving onto 6.0. I see like 5.0 the big release was like we had a new skin. A new skin, we switched over to ES6.

Gary: When did we move play button?

Matt: That was version 4 that we moved the play button.

Gary: No!

Steve: No, we moved it in version 5. No, version 4 was in the upper left. Version 5 I just doubled down on it and started getting super salty on Twittering issues. So yeah, like mostly I feel like 4 or 5 was mostly a cleanup of the code. Other than the skin, most of the changes were behind the scenes around video, like around the build process and using ES6, all that sort of stuff. So yeah, I'm curious, what should people be expecting in 6.0? So the big thing that I've seen recently is more accessibility improvements. Do you guys want to tell us a little about that?

Gary: Yeah, so for the past couple weeks and basically for a long time I've been working with Owen to improve the accessibility. So what that means is that visually it should stay the same or better for people who can see the player and use the mouse and they don't have any problems. Next level down is people who like to use the keyboard. So if you just tab over through buttons that it all should make sense, it should all work.

But then probably the most important group are users who use access technology like screen readers. So screen readers, it's very important to support them, because they want to use our players and there's no reason not to support them. It's actually really easy. Had we known how to apply them we probably should've done that, but it's becoming more of a thing now. The two big things I think that we did, we actually had a blog post on the Video.js blog about it is the outlines. In Video.js 5 in the skin we decided to turn off and had this kind of glow around the icons.

Steve: Who did that, Matt? Who would build such a skin?

Gary: And while that looks nice for people who have low vision or something, the focus on the buttons doesn't necessarily provide enough contrast, so you can't necessarily tell which button is currently focused. So if you're just tabbing through it's like well where am I looking at? I can't tell. So we brought back the outlines. The other thing is we changed around how the volume menu button worked.

So we had this thing where the volume menu, the default volume control, was a view toggle that when you clicked on it muted or unmuted the player. When you hovered over it it opened up a volume slider.

But because of the way that it was written where it was basically in the DOM it was a button with a slider as a child button, screen reader technologies couldn't really interact with both.

You could basically only make it interact with one or the other, and so it was not a very good experience. Even if you were just tabbing through it wouldn't work quite right either. So we got rid of that and instead we just have basically two separate buttons, a button and a slider control that are siblings of each other in the DOM, and we just used CSS to make it look like it used to, so for visual users it looks the same.

Steve: That's cool.

Gary: Yeah, and we just spent a lot of time just fixing all the accessibility issues. So we have, I don't know if we're necessarily WCAG compliant, but we're definitely getting there.

Steve: The other one that I'm really interested to hear more about is the new middleware component, because I remember like in 5, 5 might've been when we introduced the source handler concept.

David: That is Steve, you're right.

Steve: I think so, yeah.

David: Yeah, that's definitely where we added it.

Steve: The source handler was this mechanism to kind of allow us to support adaptive formats like HOS and Dash using a lot of the same code between the Flash player and the JavaScript player. And as I understand it, middleware is maybe a better approach to this, but I should let you guys kind of explain it, because I don't fully grasp it myself.

Gary: Right, middleware were kind of thought up because we got some plugins that wanted to take some kind of source, for example if you had an API for a catalog that you had video IDs associated with video. And now you can make a request to the API that would give you back an actual URL. Depending on the browser or what you give the API it will give you back HLS or Dash, or just MP4, or maybe an FLV, and one thing that we we're seeing is that with techs and source handlers you couldn't really have a source handler that used another source handler, so you wanted to use the service.

So what we have currently implemented is a plugin, so basically you'd initialize the player, and then you'd make the request, and the plugin would basically make the request for you. The reason for that is because we wanted to have Video.js go through the source selection algorithm and the source handler algorithm and be able to pick. So if we got a Dash source it'll choose Dash, or if we got HLS it'll choose the HLS.

But the other functionality that this plugin had was more of the stuff that source handlers and techs themselves were doing. So this plugin really needs to be a tech or a source handler itself, but there was no good way of having it delegate to any other tech or source handler.

So the middleware was designed to come in between the way that Video.js plays back the video and the player object, which is where outside users of Video.js interact with the player.

That's where like the play and pauses and all that. So it goes through and say you set a source, the source could now be that ID that the API expects, and it'll go through the middleware and the middleware will run and do whatever it needs to, and then it'll go through any of the other techs, the current tech will get loaded.

I guess back peddling a bit, the way the Video.js works we sort of went over it with the fact that it has a Flash fallback, it has a player, which is the main player, and techs, which are either like HTML5 or Flash, or I guess there's the OGV tech. OGV.js, which Wikipedia is working on, but basically it's a way of abstracting over the actual way that the video gets played in the browser, whether it's in Flash, or the media element, or OGV.js with a canvas, or anything like that.

Steve: It's kind of like an API translator, so it takes like the API Flash or the API of the video element and makes them look the same.

Gary: And so the middleware kind of sits in between, and the API of it, the reason why it's called middleware is because we basically have a routing table, but the routing table is in this case, unlike in Express, which is actually URLs, the routing tables are around the line cuts of the video.

So for example, we could have an MP4 middleware that runs whenever you load an MP4, or you can have say like this plugin is called foo, you could have a type of say like video/foo that sources the ID and it gets invoked when you give that source to Video.js. It goes gets it source, which is say an MP4, then it'll give it, hey actually what I want you to actually play is an MP4, and then it'll go through and see hey, do I have any MP4 middleware? If yes it'll go through, and then eventually it'll get final source, and then it'll do the normal tech selection.

Steve: That's pretty cool. So basically

You have something you can set as source, and somewhere between setting that source and that source getting to the video element, basically anybody can develop this middleware that will do a bunch of smart stuff in between, and it's completely open ended.

Whereas previously source handlers were a little bit more prescriptive. You had to like specifically say like this is going to be for HLS or Dash.

David: I see middleware as kind of like one of the problems that I think we constantly have with video playback is whenever you're trying to do something interesting you end up in a situation where you have like an analytics system that wants to talk to the video element, you have might have advertisements that want to talk to the video element, you have like some player code that wants to abstract over problems with the video element, you have an adaptive streaming library that's letting you play Dash or HLS, and figuring out the right ways for all those things to talk to the video on it without breaking each other left and right is like a huge problem.

So source handlers were our solution for allowing adaptive streaming libraries to hook into Video.js and talk directly to the video element. Middleware is kind of one layer above that, and it lets anybody inject some logic and do some transformation on almost whatever they want, right?

Gary: Yeah, currently in what we have in master it's a bit limited, but one middleware that I wrote recently is a playback rate changer. So basically when you load the middleware it gets bounds to any video you play and it intercepts the set current time and get current time, and also the duration methods between the tech and the player, and if you set your playback rate to say 2.0 to 2x it'll actually divide the current time of duration too.

So that means that if you have a 20 minute video, this middleware makes it so that your controls, your UI, would say that you're actually going to be watching for 10 minutes of real time, which is nice.

Steve: Yeah, it's kind of neat. I guess another real world example is a lot of people are experimenting with ad insertion into video streams. So as an alternative to client side ad insertion, you may actually just, similar to broadcast television, shove your ads into the video content itself, and in that case you have trouble building controls that look like what people people would normally call video player controls.

Most of the time you don't actually see the ad duration in the progress bar if you're looking at a video player that has ads. You just see a little marker or something like that. If you're looking on YouTube that's what you'd see. So it's not terrible to have like here's the span of the ad, you can actually see the full duration. It's different than what people expect, and meeting people's expectations for UX is an important thing.

So middleware would let you actually transform the timeline in a way that you could make those ad breaks look like markers, and then when you get into the ad you can expand it. When you come out of the ad you can pull it back.

Gary: Whereas right now a lot of ad libraries just create a separate control bar and then swap between them.

Steve: That's what we've done so far for a lot of ad integrations with Video.js, and that doesn't feel as smooth as it should, you know? It looks reasonable, but it's probably more moving parts than you wish you had for something like this.

Phil: So I still have this burning desire to build my own video streaming protocol, this burning desire that's been one of things, we talk about it every time, and MPEG SASH go look it up.

Steve: Phil, this is going to be the death of you man. It's bad.

Gary: It's definitely the reason they don't have me at MPEG meetings anymore.

Steve: You shouldn't have started that rival conglomerate. That was the problem.

Phil: So we've got source handlers, right? We've got the new middleware stuff. We've got advanced plugins as well in Video.js 6, right? So which of those bits do I need to build when I eventually build my whole own protocol? I've written a 30 line sash player that literally just is media source extensions, it's terrible. But what bits do I need to build to have a Video.js sash?

Steve: So I work a lot on videojs-contrib-hls, which is like it's a also adaptive streaming library, and I think, this is a very subjective question, but I think the direction we're heading is source handlers give you access to the video element directly, and I think building your adaptive streaming logic as like a video element, sectioning that off and working with directly the video element is a good starting point, like it's a nice abstraction.

So you have video element, you have your MSE stuff and you keep that together, and then you can use source handlers and middleware to attach it to Video.js so that it then feeds back into the rest of the ecosystem.

That's one area where I think source handlers did kind of do a good job is that they allow you to then make the couple tweaks that you need to to expose your own magical custom protocol just like an MP4 shows up, right?

I think middleware actually offers some intriguing possibilities beyond that, but you can get 90, we can get everything done that we've thought of so far with source handlers, and we'll see what cool stuff we come up with nowhere.

Matt: Nice. So something that we've talked about throughout this podcast so far has been kind of one of the big use cases early on when HTML5 video support was pretty low was Flash being able to fall back and work in older browsers like IE8. So I think it's going to be a big shock to people that it's moving out of core. So that's huge news, by the way, to anybody that didn't know that.

Flash is no longer going to be baked into the core Video.js player.

So what exactly does that mean and why is what we're going to see a lot on the issue tracker very soon.

Steve: Yeah, I think, well so what does it mean? Really honestly it doesn't mean too much if you want Flash. So like it's now in a separate project, but it's like a script that you add.

Gary: You just add it like include Video.js, include Video.js Flash and it continues working.

Steve: Right, so if you're in that camp, which oh God, I hope you're not, you can do it and it's not that bad.

Gary: We're in that camp.

Steve: Ah! But if you are one of the many happy people who don't need Flash in their lives anymore, you don't have to do anything to get rid of it, it's gone. So if you just start using Video.js 6, you don't have Flash. There's nothing there. All the skeletons have been removed from the closet.

Gary: I think the focus of Flash has been waning a lot in recent years, particularly with Apple's influence and mobile devices.

Matt: Well now Google.

Gary: Google too. They're really pushing it.

Steve: Okay, so yes, they've written some blog posts, and I believe that the word on the street was 1% of users had Flash turned to click to play.

Gary: I think they changed it, 56 I think has 100%.

Steve: Oh really?

Phil: I have two clicks to play now.

Steve: Two clicks to play.

Phil: I have my thing I used to have, and now I have to click through the extra thing as well. Really annoying.

David: I guess you're right. So what's funny is I don't encounter Flash enough these days to actually know the status on that.

Gary: And it's also weird because they still have their intelligent thing where it's like if Flash is important, whatever that means, it'll magically let it through.

Steve: I always thought that was a good idea. They've been like intentionally ambiguous, which leaves everyone who uses Flash in like this horribly nervous state, which is probably good, right?

Gary: It's really weird, because I've seen IMA ads where the page loads, you see the little gear that says, "Click to enable icons", but then a split second later the Flash ad starts playing, which I guess that is exactly chrome's detection. It's like this likes an ad, that might be more important. We should probably take it.

Steve: The most important even. They're so good, the Chrome team.

Gary: It's really interesting, because I've been browsing without Flash for about a year now with like a click to play thing that wasn't the native one so it never got overridden, and

It's fascinating the places you find Flash. You go to a random login place, and hang on, there's like a Flash click to play hidden right at the bottom corner of this screen.

What the hell is that? It's all over the shop you just randomly find chunks and little bits of Flash there. Obviously someone is doing something. I think my favorite one is Verified by Visa actually tries to load a Flash plugin.

David: I think it was like probably six months ago, but I was flying JetBlue and I tried to get to their website on my mobile phone and their whole thing was built in Flash. It was so bad.

Gary: HBO GO just recently switched from a Flash only website to like HTML5.

David: Really? That's nick of time. That's what I'd say.

Phil: Very, very quick side note on my favorite Flash fun I had last week. For those of you who don't know what PCI DSS is, the industry standard for credit card monitoring.

Matt: Oh right, that.

Phil: I fill one of those out every year. My vendor, their website only works in Flash, still.

David: At least it's not IE only. Like that used to be, before it only works in Flash that's what it used to be is like you must load this in IE8.

Matt: Well at least Video.js is now no longer contributing to the, well, will soon no longer be contributing to the Flash problem. But as we're moving, as 6.0 comes out, as we get these middleware accessibility improvements, advanced plugins, and Flash dies, what is the next big thing? Like what do you guys think is the 7.0 push?

Gary:

Middleware I think would still be a big thing, because it's brand new now, and I think by the time 7.0 is out we'll actually have a much greater idea of the power of middleware.

David: Yeah, and I know Gary brought this up when we were talking about this beforehand, but I think one of the big pushes for us for 2017 will be figuring out how we're going to tackle this whole like media source extensions adaptive streaming world in Video.js. We have projects that do it. So we have videojs-contrib-dash, which plays back Dash video, and contrib-hls, which plays back HLS. And the question is like is the world in a place now where not having that be part of Video.js we're missing today's problem, right, which is playing more advanced video formats?

Gary: That's probably several projects, and the question is like should we shift them with Video.js? It seems like the world is moving towards yes, because it seems the adaptive streaming formats are becoming so popular, and people more and more are starting to expect like I want to grab a player, and I just want it to play any format I give it.

David: Yeah, yeah wait, that's a lot hmming, Steve. Do you have some thoughts on that?

Steve: No, I think it makes sense. As always, a tough call which features to include and which to not. One of the lines that we always drew was does the video element support this thing? If the video element supports it, then we should support this in both HTML5 and Flash, and then everything else on top of that should be a plugin. We didn't follow that exactly, but that was the mantra, right?

David: Yep, I always liked that mantra, still do. But this one is really funny because it's like, well, it sort of does and it sort of doesn't.

Steve: Yeah, some of them. Some browsers do support those, like Safari supports HLS, Edge supports Dash.

Gary: And HLS.

Steve: And HLS, yeah. So you could make a strong argument there, I feel like.

David: And I also think like the other interesting problem that's happening is if you look at any of the libraries that do adaptive streaming now, they're all kind of fiddling around with the idea of just supporting like essentially any segmented media format that you can write a manifest parser for.

Gary: They're all kind of converging on all of the parts to do one thing.

David: Which actually is awesome. I can't even think of a time, well maybe like back when we were always doing FMS, but it's hard for me to remember a time where like it was like one video format and like everything just worked, but it almost seems like there's a very glacial movement towards let's just use fragmented MP4s in reasonable segment sizes and we'll figure out some text file to specify them, and maybe it's the magical work at MPEG sash, right? And then that'll just work. You could actually write like a reasonably sized library that does that and it wouldn't be an explosion to Video.js to include it, right?

Matt: So it sounds like you heard it hear first, the next big initiative for Video.js is MPEG sash. I'm fairly certain.

Gary: Another thing I think we need to look at is custom builds of Video.js, because we've been hearing more and more people who they want Video.js but they don't care about captions at all, or they want Video.js, I mean they should, or they want Video.js but the way they're appointing it they're never going to need a control bar or any of the UI stuff.

So it'll be nice to look into being able to provide custom builds so the user can build, okay, I want these components. Because if you want Video.js for like the plugins and the techs but you don't need the control bar, the control bar makes actually probably the majority, the UI components, make the majority of the file size of Video.js, and so you could slim down your deploy quite a bit if you didn't need that.

Steve: I've actually been thinking a lot that it would be cool to pull core out of Video.js and have that as be, because then you could use that in things like web components or React components and kind of build.

Gary: Have everything be plugins?

Steve: Yep.

Gary: That is actually something that we we're thinking with middleware, because, and be advanced plugins, because that way the middleware works is that you register a factory method that when called it takes a player, and then when the source is set the middleware gets the tech.

So the middleware gets both the player and the tech, and then with the new advanced plugins, which have more lifecycle events, you could, using middleware, you could add stuff to the tech, more functionality, say attach a source handler or something like that, and then on the player side add a plugin that adds the corresponding functionality as an API to users.

So theoretically most of the things in Video.js could be pulled out to do plugins with middleware.

Steve: Plugins all the way down.

David: That's crazy.

Gary: Maybe that'll be 7, have everything be plugins.

Steve: You just like skipped like version, just do it like React style and just like jump to like version 18 or something at that point.

Gary: 600.

David: The core becomes an empty file. It's always been our dream.

Matt: Easily maintainable, but everything would just move to the grunt file at that point.

Gary: We're switching to vanilla.js.

Matt: No stop. Alright, we've run up on our time at this point. So I just wanted to say thank you, Gary and David. If anybody out there has a very specific burning question around Flash, gkatsev and dmlap on the issue tracker.

David: Not around Flash.

Gary: Actually just come to our Slack. Slack.videojs.com

Phil: Be sure to ask about the positioning of the play button.

Matt: Next episode. Okay, well anyway, thank you so much everybody. I really appreciate you guys joining from Boston, and of course Phil from London, but you're here every week, or every episode.

Steve: Every week?

David: Yeah, now it's a weekly show.

Matt: Alright, well thank you again everybody.