Building an API for many devices and choosing native vs. web apps at Plex.tv
Interviewed by Christophe Limpalair on 01/18/2016
Plex is used on many different devices and platforms to stream all kinds of content. They've had to decide between building native apps versus web apps, and whether that still makes sense today or not. They also had to build a Rails API that could support all of these different devices and 21,000 requests per minute. On top of that, they have unique transcoding and streaming challenges because media is stored locally and not on the cloud. This episode explains how they do it.
Links and Resources
How is 2016 going? Do you have some exciting things going on at Plex this year? Do you expect some big improvements or changes?
Excited to be here talking with you Christophe. 2016 is going to be great for us. Can't spill the beans and get in trouble, but really exciting things happening.
Can you tell us a little about yourself, how you got started with Plex and what you do there?3:09
I got started with Plex because it was started by some of my friends I had worked with in the past. When I saw they were working on this media, fun, consumer focused product and that they were really going to be able to turn this into a real thing, I jumped at the chance to join them. I've been there 5 years now.
How long before they started getting traction?3:53
They probably would say it was a real thing for about a year before I joined. As a hobby, it goes probably goes back a couple of years before that.
What do you do at Plex?4:23
Over the years I've done a little bit of everything. We started as a very small group. As it has grown, my main role is one of the directors in engineering organization on cloud things. We have a Rails API, servers and clouds, some DevOps and for good measure, the Roku client, which has always been a pet project of mine.
Can you tell us more about Plex? What can people do with it?4:58
The simplest explanation is, "Your media everywhere." We feel like everyone has media on their computers today; photos, videos and music, Our goal is to give you "one place" for you to organize your media and and then empower you to access that media anywhere. You can pull it up on your computer, your phone, another computer in the house, on your tv, at work when you're 3000 miles away. Your media truly everywhere.
So how does that work? Do you have one central location where users upload their videos, sound, music or whatever, or is it stored locally on their computers?5:47
Yes, stored locally is the main model. You download the Plex media server on your "computer." It could be a laptop, old Mac mini you've tossed in a corner. Basically, anything capable of running software can run a Plex media server.
The reason we chose that model is that that's basically where the media lives. That makes us a little different from other media people, especially my being a "cloud person" by my saying that the first thing you need to do is install some software. Computers have a bunch of "time capsules" with photos and music on them, so the simplest way is to have a Plex media server running nearby that data and then expose it everywhere.
So you have a Plex media server and the clients connect to the server. So that's the main difference between the two. The client doesn't actually store any of the files except when it's transferring, but the server has all the files?6:58
Exactly, the clients have gotten smarter over the years, but the idea is that they can be kind of "dumb" and that allows us to run clients all sorts of places. There's iOS and android, for sure, and then there are smart TVs, which aren't always as capable as as a phone or computer. Play Station, XBOX, pretty much anything can act as a client in that model because the clients are fairly simple, in that model, just consuming the media.
That sounds great if you're nearby, but what if you're in a different country or many miles away? How do you avoid latency issues?8:25
There can be problems. You will be capped by your upstream bandwidth. If I am thousands of miles away on vacation, for example, I can stream, but if my upstream bandwidth is terrible, it will be a limiting factor. There are a few ways around that. You can choose to sync that content to your phone before you leave or to the cloud with Cloud Sync. I can stream using my personal cloud storage. If I have Dropbox, Google, Amazon, I can choose to store an optimized copy in my cloud.
Are you envisioning other ways to help with that problem? How do you work around the capped upload?
Christophe: I have had to manually adjust quality before because it would keep buffering. To be fair, I wasn't streaming in the same house, it was across different cities because I was using a shared library with my brother.
Looking forward, we are investigating ways to make that better. In your example, you were manually stepping down qualities. It's possible that we could start optimizing that for you so the user wouldn't notice. It's an area we are actively looking at.
Do you envision moving from local host to uploading to your own cloud and using a CDN in between? Would that be way too expensive?11:20
That is certainly something interesting to think about. For now, the most compelling reason not to be in the cloud is that the data is not in the cloud. Most of our media collection is stored at home anyway, so it makes sense for the server to be at home. That's something we're keeping an eye on. People are increasingly storing media in the cloud.
When you install that locally, how does it work? You call it a server, but how does the installation work?
Basically, the media server itself is a software package written in C++ which will run on just about anywhere; Windows, Mac, Linux,and Nas platform running on different flavors of ARM. You basically just install that and run it.
It opens a port you can connect to. How do you secure that when people share their libraries?12:47
Yes, when sharing enters the mix, cloud becomes more a piece of the equation. At our main site, Plex.tv, we host the API. By sharing, basically you install the Plex server on your computer and then it communicates with Plex.tv and tells it that it's a new server, who it belongs to, and that it's available at a certain IP address port. That connection information is stored at Plex.tv even though none of the content is. So now, the person who is sharing from a different IP address, talks first to Plex.tv. Then, it brokers the connection between your server and the one who's sharing and serves as authentication. It's a token based authentication scheme.
Christophe: Right, I actually checked the network connections before this interview and I could see where there was a primary server connecting and shortly after, there was something called Titan. Titan had a different address it was connecting to and it was switching from the primary to Titan. You told me that was probably the server I was connecting to.
Can you manually name it?14:45
Yes, out of the box we try to set some defaults, depending on the platform, you might be able to ask the OS for a "friendly" name. You could always give it a name that you want. I would guess that someone shared a server named Titan with you.
Yeah, app.plex.tv has an amusing history, but basically that's just the front end for a single page JS application, so the shell of it is really small. it just exists to get that authentication token and then pull down the main app. Then, it uses the API to get a list of servers. Then, it talks directly to the servers. It authenticates and checks out the content in the library to get that first class media browsing experience.
What's powering the Rails API?15:57
Nothing too exciting to be honest. It has evolved. When it was first created, Plex.tv was called MyPlex and was a really simple Rails application having servers and choosing to share them. It has grown up a little since then, but it's a Rails stack using Sidekick with Unicorn for workers.
Where are you hosted?16:50
We got set up with Engine Yard (cloud orchestration platform) years ago and have been happy ever since. It's built on top of Amazon so it's EC2 instances under the hood, but a little simpler management.
Can you share the kind of stats you have running on those (number of requests)?17:17
On one of the more common endpoints, looks like we were getting about 21,000 requests/minute.
You have JS and Rails API for desktops and laptops. Do you build a native application for cellphones that communicates with the Rails API?18:10
The client applications generally start by talking to the Rails Api, talking to Plex.tv (that's mostly to discover what servers are out there). After that, your server is talking to Plex media mostly. It's mostly all native on those platforms, not a lot of shared code.
Why did you choose to do that?19:26
It has changed over the years, but it was mostly because we were putting user experience first. We wanted a first class experience, and that meant going native. Rather than building one app and having it run on twenty different platforms, we would rather build an app on each platform so that when you're on your iPhone, it feels like Plex but still feels like your iPhone. They feel platform appropriate.
More recently, we have accepted the wisdom of seeing what we can share between those because that argument is mostly about UI. We have experimented with taking some of the complicated bits and seeing if we can write those as a shared library.
With your experience, would you still recommend taking the native approach? Would you do anything differently?20:55
Honestly, I would probably do it the same way for a couple of reasons. First, if you were just talking about android and iOS, it's sort of a "known arts" on how you might build a shared library usable by both, but with Roku and smart TV, they're different languages and more difficult to share. Knowing we would only be able to share on a subset of platforms anyway, it makes sense to do it that way.
Secondly, it's difficult. For longterm maintenance, it would be a big win, but there were some big headaches building those shared libraries. If we had tried to do that from day one, it would have taken us much longer to get a nice app out to the public.
Did you create different teams to build these different apps or did you adapt and learn as you worked?22:02
We tried to "cross-pollinate" for many reasons, but finally, there was a team per platform for the most part. The teams communicate as much as possible.
It seems like we're really making advancements in "bridging the gap" between the native feel and web-based feel where we have things like PhoneGap, I don't think we're quite "there" yet. Will we get there?22:30
I'm a skeptic, but I want PhoneGap to be an experience that feels native; write it once, run it everywhere. I love that people are working on it.
How much of it is native? How much can you keep on the device? Every time I've tried to build an application and connect it to a cloud, for example, you feel that latency between the two. How do you make it fast?23:15
It's mostly native. We're getting data over the API, but that's inevitable for us.
Can you give an example of what kind of data?23:53
Let's say you're browsing the library but mostly returning that as XML. You have an XML representation of your photo library with rich metadata. We try to tune how much metadata and how big that XML is based on where you are in the browsing hierarchy.
You have 2 options: Send everything so future browsing is really fast, but the initial load is painfully slow, or, as we have done, make the initial request small to render that step of the UI; a kind of "level of detail" approach.
How much of it can you actually cache?24:43
We could do more. We've been nervous about caching too much. A lot of library stuff won't change, but just enough does (things like ViewStates). If I watch it on some other clients and then I'm pulling it up here and I want to see the next watched episode, refetching that list fresh, then I might get some stale data there. We have erred on the side of caution.
You cache that natively? What kind of caching do you use for web browsers?25:25
When they're talking to the Plex media server, not a lot of caching for the actual application itself. We try to let the browser cache that.
When we were talking about connections and switching from the primary to the other server, I noticed that some of these connections were through Web Socket. What is Web Socket?27:30
One simple example is when you add a new library within your Plex server. You're inside the web client and you're managing the media server and you say," Oh, I got some videos in this folder, and the media server goes off, then it's scanning all these things in and finding metadata and doing all this work rather than the web client continuing to make pulling requests every second and milliseconds. It's able to have just one Web Socket open to the media server and it updates to it. It allows the web client to have a much snappier interface and get those updates in a push way rather than having to pull.
What would be some other use cases for something like web socket?28:50
The canonical example is the chat programs. If you were to go on a web socket tutorial page, I'm sure they would have you writing a little chat client.
You can just go in your web server(I think you use NGINX) and configure it to have that?29:17
Yes, for the media server, it was an adventure because the media server doesn't use NGINX in the cloud, but in the media server it's bespoke built on top of Boost libraries. It was harder than we thought it would be.
Chris: It's never that easy when you start implementing. You always have a different use case.
I am curious about the transcoding especially since it all seems to be happening locally. Can you give some details on how it works?30:00
We use an open library, FFMPEG, which is an awesome open source library for media processing, especially video. Basically, the Plex media server comes with our variance of FFMPEG to do that on the fly realtime transcoding.
It's especially interesting to us because of the variety of platforms. I might have some fairly innocuous video format on my machine, but if I'm trying to stream it to iOS, Android, Roku, Visio, Xbox, and all these things, they all have little tweaks on what they can and cannot handle. Mp4s are fairly safe but some are optimized for streaming and some are not. Mkvs are pretty well supported but not quite as well. Wmv files are pretty Microsoft specific these days.
The Plex Transcoder gives us the ability to make sure users don't have to worry about that. I don't wish it on anyone to be forced to understand all those formats.
We want to be able to say,"Whatever video files you have, just drop it into Plex, and we'll worry about what needs to happen for it to be consumed."
That sounds like another tough challenge because if you were storing all these videos on your own servers, then you could have machines specifically optimized for transcoding. As it is, you're doing it locally on people's machines. You don't know what kind of CPU they have, they may have really slow machines. Is that a problem?31:40
It's a blessing and a curse. On their engineering blog, Dropbox talked about how they pre transcode the first few seconds of every video, and then as you play it it has to transcode the rest. That's a lot harder without knowing that you have infinite computing resources on the media server. There's no "trick". It's something we constantly think about improving. The main thing is to be aware of the power of the computer.
If you're running on an ARM based NAZ, we'll say you can't transcode. We'll let you remux. It's difficult to understand. Remuxing means that you unpackage the container of the video, but you would copy the actual video stream as is.
If you had an MKV file with H.264 video in it, but the client needed an HLS container, you could remux that H.264 video from MKV to HLS. That might be easy enough to do regardless the processor, but actually converting that video is another story. As we get into 4K and H.265 it's even more intensive. It's definitely a challenge.
Transcoding really depends on what kind of machine is there?33:55
Yes, and the other challenge is because it's your servers. On the plus side, we're not paying for servers. On the downside, we're mindful of wasting that storage space. If you've got a spare hard drive, if we were to transcode every single file, your usable space on that hard drive would shrink to 20% or so.
One of our features we launched last year was called Media Optimizer, which lets you chose if you want to pre transcode and at what quality. You might transcode the more recent videos or certain subsets, but not all.
This is really different from the other two interviews I mentioned earlier with particularly Netflix. They have a lot more flexibility in transcoding and caching as much content as possible. You definitely have some different challenges. Have you read some of their content that has been helpful to you?35:90
Definitely. They have a great engineering blog. They've been open about challenges and how they've attacked them.
Another really interesting subject with Plex is that you pre-populate metadata, right? How do you find that?36:35
The interesting thing about metadata is that there are tons of sources, and we made an early decision that we don't want to become a source of metadata. We don't want to manage those huge databases of all of the media ever created. At the same time we need to be an intelligent layer that allows you to access those. We work with the ones that have good data. We sit on top of the existing libraries and choose the best features from all of them.
Hypothetically, in the music case, rather than using everything from one source, we have agents that run against each of them. These combiners look at each of those and choose the summary from one, the artwork from another and so forth, and then present that to clients.
Are you saying this is automatic?38:36
Yes. We've written those agents ourselves and made some default ordering decisions based on general quality we've seen.
How can you tell?38:53
In the album description example, you look at a few albums and it becomes obvious that one is user submitted and poorly written and then another will lack access to some database....official reviews...
Then do you pull that and store it on your machines and servers instead of having to rely on somebody else?39:17
Yes, we have that distilled version and anytime we need to we fetch the original source either to refresh what we cached or grab something if we didn't find it in our cache.
What kind of career advice would you have for someone interested in working on this kind of application or working in the cloud...?40:03
Career advice would vary depending on the context, but we've been really fortunate in having a ton of people that have found ways to work with us (just pitched in) that ended up getting hired full time. I don't know if that works at a high percentage of companies, but it's been an interesting lesson. If something interests you, just go work on it. You have two possible outcomes. One, you were interested in it anyway and you had fun learning about it, end of story, or two, You had fun, caught someone's attention, and turned it into a full time paying gig. No huge downside either way.
The other advice is, "You can't be afraid to put yourself out there, which is the hard part, but exceedingly true."
The one piece of advice I probably give to everybody (from my own life) is to not be afraid to use people you know to learn about opportunities. When I was first getting a job out of college, I had a degree in computer science and I really wanted to get a job the hard way....submit a resume on some website and have them "pluck" it out of the pile, be amazed at you and offer you a job, which, as it turns out, is incredibly difficult.
I ended up taking a job through somebody I knew. Once you get that first job, you have the opportunity to actually do good work, learn and prove yourself. Every job I've had since then has somehow been traced back to demonstrated accomplishments.
"Getting that first job can be really hard. There are a thousand ways to get a second job."
Chris: The entire time you were talking, I kept thinking about what happened to me personally. I back up what you said a hundred percent. It works. It's true.
If people want to reach out to you or anyone at Plex, how do you recommend them to do that?43:40
Plex.tv for general information. We have forums where you can get information about the products and problems. We have developers and community members helping people on those forums.
How did this interview help you?
If you learned anything from this interview, please thank our guest for their time.
Oh, and help your followers learn by clicking the Tweet button below :)