How Netflix handles 36% of the U.S. bandwidth, and how reddit handles 160M users
Interviewed by Christophe Limpalair on 07/28/2015
Learn how Netflix scales to handle 36% of North America's bandwidth during peak hours, and how Jeremy scaled reddit as Chief Architect. reddit now has 160M users and billions of page views a month, and still uses a similar architecture. See how the two companies solve different challenges.
One common challenge is tracking when things go wrong. How do you track errors in development and production? I use Rollbar and really like it. They set up a page for viewers to try it.
--- reddit section --- (Go to Netflix section)
How did you become reddit's first paid employee?
Jeremy went to the first YC startup school and met the reddit founders there. He says it was intentional because he had had a similar idea and found reddit by searching on Google. So he been on reddit for a couple of months before he flew out of Boston and was invited to a pre-party, which he thought was a mistake because everyone was either part of Y Combinator or one of Paul Graham's friends. Jeremy says he found out later that he was invited because he was one of the only people there over 30.
Anyway, he met the guys there, had lunch, and handed them his resume. They didn't really know what to do with it because they'd never hired anyone before, but they ended up chatting and being friends. They went up to California a few times and Jeremy let them stay at his apartment. Eventually they got acquired by Conde Nast (media company) which enabled them to hire Jeremy full time.
He met the team a few months after reddit launched, and joined them in March 2007 which was about a year and a half after they started.
What does reddit's architecture look like?
Jeremy stopped working at reddit back in June 2011, so some of this may be different today, but Jeremy spoke with people at reddit a day before the interview and it is still quite similar.
Everything runs through an HAProxy load balancer and gets farmed out to Python applications.
The databases are all PostgreSQL.
There is also a set of Cassandra cluster.
And a Memcache cluster.
RabbitMQ for queues.
Since he's been there, they've improved the codebase quite a bit and added other technologies like ZooKeeper.
Here's a little bit more detail on reddit's current infrastructure.
What were some technical challenges that took down the site unexpectedly?
Generally speaking, they would scale the site once they hit a point of failure.
Some of the failures had to do with the way they cached.
Another issue they had was with rendering. They use markdown, and they were using a Python markdown library that was just too slow to keep up. They fixed that by replacing it with a C library that they turned into a Python module. Then someone created one called Discount which is what they ended up using. Jeremy thinks they're still using it today but he's not entirely sure.
I know they used snudown for a while. Need to check the source code to see what they use today.
Another way they improved scale was by accessing memcache and DB with C libraries.
When they witched from physical servers to cloud servers, latency between services went from 1/10th of a millisecond to a millisecond. This makes a big difference when memcache gets thousands of calls per page load, that delay adds up. They solved this by getting larger batches of data with fewer calls.
Has latency improved in the cloud?
They switched to cloud back in 2009, so I asked Jeremy if latency was still as much of an issue today, or if there have been improvements.
Naturally latency is going to be higher in cloud environments vs bear metal, but it's a known issue that we can architecture around.
Jeremy said, at the time, reddit would scale when they hit bottlenecks. At Netflix, they proactively look for bottlenecks before users 'find' them.
While Netflix certainly has devoted a lot more resources to do this, it's also important to keep in mind that reddit only had a few employees at the time. When you are in a startup environment like that, the resources simply aren't there. This resulted in users finding most of the bottlenecks before the employees did.
They did find some bottlenecks by profiling their Python code. Here is more info on profiling from the wiki and from the docs.
With a small team, should you focus on features first and put out fires, or should you really focus on scaling?
At least put some thought into scaling, but make sure you don't prematurely optimize. As long as you use good software, it will be able to handle at least some of the initial scale.
Jeremy says: start off with a good architecture
It's really more of a business decision. In Jeremy's mind, it's better to move forward with features but you do have to find a balance that fits your business.
How much money do you lose every second that the site is down? This is easier to define for companies like Amazon and eBay, because they know exactly how much they will lose per second. It's harder for other companies like startups or companies like Netflix which charge per month. If the application stops working for a minute, they won't lose x amount of sales, but overtime users will get tired of it and switch to a competitor.
What should sites like reddit monitor?
Jeremy's general advice is to monitor application metrics.
You can monitor server stats like CPU, Memory, etc... but those aren't going to be as useful, especially in a cloud environment because you can just delete those instances if they act up. At the end of the day, it doesn't matter as much if your CPU is spiking as long as its delivering customer traffic.
Monitoring depends, in part, on your business. At reddit, when Jeremy was there, page views per second mattered the most. That's really what they focused on primarely. At Netflix, the main metric is play starts-- how often someone hits play and successfully starts the movie, and how much time elapses between the user pressing play and Netflix's services processing the request.
If those metrics dip down or spike up (depending on what you're monitoring), then you know something is wrong.
When something does go wrong with the metrics, how do you debug?
Debugging usually involved:
1) Checking if something had just been deployed
2) Running to the logs to see if there was anything wrong
Eventually, they built a system that would fingerprint Python error messages, so it would tell them every time there was a new error, and it would count them.
3) Check if the count had spiked
4) Checking monitoring on servers for spike in resources
About half the time, answers were in the Python logs. The other half was just a server being under very heavy load.
5) Checking queue length
Everything they could queue would go into a queue (incoming votes, incoming comments). Monitoring the queue length would give signs of an issue. If the vote queue, for example, had backed up, then there was probably a problem with the DB.
reddit uses key/value store for a lot of its data. Why?
Jeremy is not entirely sure why they initially chose this, but it turned out to be a great choice. It turned out to be great for scaling.
Looking at the site, it seemed to me like their data was very relational. Users have votes, comments, posts, etc.. Posts have comments, votes, etc..
Jeremy says the data is not relational in a typical sense of the word. There are a lot of relationships, and so they have tables in place to specifically keep track of relationships. But those are all storable in key/value store.
Why PostgreSQL instead of something like MongoDB? (or anything else specifically designed for key/value)
MongoDB didn't exist at the time, but PostgreSQL is just a very solid database. It is battle tested with 25+ years of practical use, "it's the cleanest code you'll ever look at", and "it's more performant than most of those other databases.", says Jeremy.
Would you still use PostgreSQL even with alternatives now?
MongoDB has a lot of negative feedback.
Jeremy says MongoDB is a special case and that he would never use it. At least in the beginning, MongoDB's developers put speed over durability which meant that writes would be marked as complete without actually being complete. The app would proceed as if the write had completed, and data would be lost. They apparently fixed that since then, but for a database Jeremy prefers durability over speed.
An alternative? PostgreSQL.
What tools do you recommend for monitoring?
If you are running in a cloud distributed environment, you need a monitoring tool that can handle this, and there aren't that many good ones out there right now.
Netflix has open sourced their monitoring system called Atlas which was designed to be used in the cloud.
People at reddit use Graphite.
When Jeremy was at reddit, they were using Ganglia, which was great for monitoring large groups of machines, but not so great when in a cloud environment where machines go in and out of service all the time.
How do you backup a large database without causing downtime?
The backup strategy at reddit is that every database has at least one read slave, and backups are pg_dumped during low traffic periods.
That worked really well when Jeremy worked there because they had a distinct low traffic period. As reddit grows more global, it will become more difficult. They will need to add more read slaves than necessary so that they can lose one for backup and not cause any issues.
How much data can you actually cache?
Caching was fairly easy when Jeremy was there. Everything on the front page is in cache because that's what most people are looking at, resulting in a hot cache.
Older pages were not cached and resulted in slower load times, which actually caused issues at the time because Google would crawl these old pages so they had to rate-limit Google's bot.
Now, Jeremy thinks that they lock pages after 6 months or something like that, and so the page does not have to be rerendered.
Why did you choose Cassandra?
They used MemcacheDB which was working really well...until it stopped working really well. There was no easy way to scale it without doubling the size of your cluster and having a lot of duplicate data. They looked for a more scalable and distributed key/value store. They found Cassandra to solve this problem.
It was essentially used as a cache at the beginning, now it's used to store unique data.
Cassandra also has very fast negative lookup, so reddit uses it for voting. When you load a comments page on reddit, and let's say there are 1,000 comments, the app needs to know how many of those you've voted on. Chances are you've never voted on any of them, especially if you've never visited that page before. Cassandra is very nice in this use case because it can return the negative results very quickly using a bloom filter which can see that there's no data without having to pull any data.
They also implemented a smart feature-- they would record the last time a user interacted with reddit (like voted, commented, or whatever) in the user database and if a post was newer than that date, it was impossible that you had voted on any of the comments so that would get skipped entirely. That was a huge optimization.
What other data did you use Cassandra for?
It had caches of listings, so they would render the listing page for, say, their programming page. It would store the top 1,000 items. There was a queue, and every time a vote came in for programming, they would recalculate these links.
What techniques were employed in "future-proofing" the architecture to ensure scalability as usage increases? Asked by: fukitol-
They didn't really put a lot of thought into future-proofing, instead they thought: "how can we make this scale as far as possible?"
Using the key/value store was one way of doing that. Also making sure there was no state saved in the app servers, and that they were totally independent. This actually brought on another scaling problem because you have to store your state somewhere and that cache has to be resilient enough.
"Share nothing" and think of servers as being easily replaceable. Focus on making app servers horizontally scalable.
The database is not quite as easy to scale up, but having support for read slaves helped a lot. As load goes higher, you can bootstrap a new read slave. After a while, they made the master database only for writes.
When does it make sense to open source your app's software?
Jeremy says that this is an interesting business decision.
At reddit, they realized that the value was in the community, not in the software. They wanted the community to be able to see the software-- so they could potentially help out, and just to be more transparent and answer people's questions. It's also a way of giving back to the community...since they were using so much open source, it made sense for them to also open source. Since they had a lot of traffic, sometimes they were the biggest website running certain open source tecnologies, and people wanted to know how they did it to learn from reddit.
For example, they were the biggest site on Pylons for a while, so people wanted to know how they ran Pylons in a high scale environment.
At Netflix, the general rule was: if it has to do with programming or computers, then it should be open sourced. If it has to do with movies, then it's not.
--- Netflix section ---
Why did you leave reddit for Netflix?
Jeremy had been with reddit for 4 years, and he was the only person from the original group still there.
He left at 4 years, 4 months, etc.. and it just felt like a good time to leave. They had hired a whole bunch of people that were able to help out, and he wanted to move on to bigger challenges.
What does the Netflix architecture look like?
The secret is magic.
Well close enough... it is a microservice architecture.
Here are more explanations: Introduction to Microservices and Adopting Microservices at Netflix: Lessons for Architectural Design.
Jeremy says that this is really where a lot of the reliability comes from.
Netflix used to have a monolithic application, but it got broken out into services.
The advantages that services provide: You can scale, develop, and debug them independently. This is very helpful during outages because they can zero in on a particular service that's causing a problem.
There's also the general architecture of what Jeremy likes to call "doing it in threes". They use three availability zones per region and have multiple regions. They have designed their services to be able to withstand the loss of an entire zone. Obviously, this makes it far more reliable. They know this works because they do extensive testing.
They test extensively in production with Chaos Monkey, which randomly kills instances and Chaos Gorilla which simulates an outage of an entire Amazon availability zone.
If you don't do testing, you can't possibly know how reliable you are, so that's a really important part.
Why isn't everyone running Microservices? Why is reddit still monolithic?
There's a lot of overhead with Microservices.
You need to have a platform to do it upon. What Jeremy found in talking with other companies is that they spend roughly 25% of their engineering efforts on their platform and running their services.
That's a big overhead for smaller companies. That's definitely a main reason. Bigger companies take a while to get there and it takes concerted effort.
This is a pain point that Jeremy and the team at CloudNative is working to solve for companies.
How do you make sure Netflix scales to handle spikes?
The trick is in auto scaling.
There are 2 kinds of auto scaling. There's reactive auto scaling, and there's predictive auto scaling.
Reactive is the kind that amazon provides: You can program it to spin up more machines when a certain metric gets too high.
Predictive is something Netflix built for itself. It essentially predicts how much traffic it expects to happen, and scales up ahead of the prediction.
The quicker you can spin up machines, the less overhead because if it takes 5 minutes to boot, your algorithms need to scale up 5 minutes before the load is actually required.
So you can precisely predict when your heaviest loads are going to be?
There's a lot of complicated math behind these predictions but it's very smooth. People's watching habits are very regular.
I could see that...I usually have a specific time for watching shows ;). We are creatures of habit after all.
Saturday morning has a pretty big spike because of kids getting on to watch shows.
There's also a little spike every hour. That's because traditional TV has shows that end right at the hour mark, and people will jump on Netflix after watching something on TV.
The really interesting things to predict is around Superbowls and Oscars. The moment something like the Superbowl ends, Netflix usage skyrockets because people are already on TV and probably have friends and family over they have to entertain... You know, having a good time ;).
How do you boot machines faster? What is baking?
At Netflix, they believe in fully baked images.
That means that when you turn on a machine, it's ready to serve traffic. It doesn't have to download any configuration, it's ready to go.
It, obviously, helps with boot up time, but also with homogeneity which means that when you run a cluster of machines, you know that they are all identical. You don't have to worry about whether one got an update yet or not, and it makes deploying code easier too.
The biggest downside to it is rapid development. You have to wait for the bake to finish. You're essentially taking the startup configuration time and putting it all 'at the front', so it does slow down development a bit.
Would you only use baking at a Netflix scale?
Baking really makes sense for everybody, but there is a big trade-off.
So it probably makes a lot of sense early on in a company's life. But at the same time it does because you get the operational benefit from it. There's definitely a middle ground.
Jeremy says that Netflix is potentially moving towards using containerization to speed up the whole process.
Do you foresee any limit to how much you can expand the current netflix system before needing to do a major overhaul? Asked by: mustang2002
There are always a few tweaks you could do, but the Netflix system is pretty robust and Jeremy doesn't think it will need to have major overhauls any time soon.
Follow up question- Is that because of Microservices?
Yes. Because if any one service is having scaling issues, they can just re-architecture that one service to meet their needs. The whole system doesn't have to be modified.
Thanks to Rollbar for sponsoring this episode!
I mentioned using Rollbar at ScaleYourCode in the intro, and I'm making all of our error handling use it now.
Here's why: say a potential sponsor tries to submit a request but our 3rd party email API isn't working, without Rollbar, we probably never see it and lose that business.
What about catching errors in development? Or errors that make it to production that users are running into but you don't even know exist?
Rollbar will notify you when it happens, it will tell you what line of code caused it, which deploy caused it, who deployed it, and so much more.
Go to try.rollbar.com/syc. It took me 5 minutes to set it up with Laravel (seriously, I timed it), but it works with any language and any tool you use. Seriously, you could have it notify you in Slack, Pager Duty, Logstash, or whatever else you use.
Thanks for sponsoring the show!
How important is the type of tracking Rollbar provides for companies like reddit and Netflix?
It's actually really important. In fact, they built a similar system at reddit. As we talked about earlier in the interview, this system fingerprints error messages and tells them how often it's happened.
Jeremy was actually really excited to learn about Rollbar because it's the commercial version that solves a need they had.
It was super handy for them at reddit because they could see the type of error that was suddenly increasing, what caused it, and what line of code caused it.
At Netflix, they look at it a higher level than errors coming out of code. Generally they are not looking at errors in the code unless they are trying to debug a particular problem. They are monitoring more high-level application metrics. That comes from both the server and client sides.
They have a nice advantage at Netflix in that they control the client side. That means they can put a lot of instrumentation directly in the client, and that feeds right into the Netflix application.
Since it's coming from the client, it's taking a different path into the Netflix system, and so they can have alerts on the comparisons of the same metrics on the client and server sides to let them know if there's a mismatch.
Play starts is one of those metrics they closely monitor. They can tell when someone hits play on the client side and when their server receives that signal. That enables them to see if there are any delays.
How do you handle so much data being passed around?
They like to joke that Netflix is a logging service that happens to play movies. There's a ton of data going back in the server, and luckily a lot of that is just handled by Amazon.
The data goes in to one endpoint for client-side logging and then the other endpoint is for the API. The API is basically the front door to all of Netflix. The API service is really good at scaling. These services represent a good chunk of all the services at Netflix.
So a lot of data is going through the API and then spread out to the different services that need it.
So do you have two different layers of load balancers?
Yes. They have the ELB load balancers for when a client hits the API. That directs to Netflix's own internal Layer 7 load balancer called Zuul. Zuul routes on request type and has the ability to write small chunks of code to do more advanced routing. That will route the request to the appropriate API machine or potentially backend servers.
The API machine will do processing and reach out to all of the services behind using an on instance load balancer called Ribbon. So it will hit that load balancer and that will farm out to all the Microservices behind.
The API is really interesting because it can enable rapid iteration for the UI teams. They can write small chunks of code to create their own API endpoints for their service. So they can say: "Here are the 20 things I need to know", and they will create a single API endpoint that returns all 20 things in one request. Then that code actually runs on the API machine and it will farm out the request and gather all the data and bring it back and send it all back in a single answer.
How many machines are we talking about, in general?
Thousands of machines. The API farm itself is thousands.
What's the secret to serving so much video content?
Netflix cheats. The video itself doesn't come from Amazon. It comes from a CDN. It used to be form Limelight, Level 3, and Akamai. Now it just comes from the Netflix CDN itself, called Open Connect.
These are machines that are highly optimized for delivering large files, and that's it. They're so optimized for that, in fact, that Netflix still uses Akamai for all the small assets.
Essentially, they are physical machines with lots of disk drives spread out all over the world in various tiers. In one location, there may be a data center with a couple of racks that have the entire Netflix library and then there might be another data center somewhere that has maybe 80% of the most popular content. If the client is watching a video that is really popular, it will be directed to the closest CDN. That means that the longer tail content you're trying to watch, the further you have to travel to get the content.
The client has intelligence built into it that determines which place to grab the video from. So the API will send back and signal that the content this person is trying to watch is available on, say, 10 different CDN servers. At that point, it's up to the client to pick one that it feels is best. It determines that by testing them all, determining latency, and continually running these tests throughout the video streaming process. It will automatically switch between the different CDNs and between bit-rate levels depending on those factors (and probably a few others). That's the whole 'adaptive streaming' concept. It's always performing tests and determining what sort of bandwidth it's able to get in order to find the highest possible quality at the lowest possible latency.
So CDNs is really the magic behind serving video content?
Yes, for static content, absolutely. Live streaming is a different story.
Your CDNs are run on Nginx, right?
What's the main different between CloudFront and Akamai
CloudFront is just another CDN but it's not optimized specifically for large video.
What advice would you give to developers looking at building a video streaming site from scratch? Asked by: ChefLadyBoardee
If you're just starting out, there are options out there for you to leverage (like I mentioned earlier in the interview with YouTube and Vimeo, for example).
If you want to build your own solution, look at open source software like Open Connect of Netflix and see how they do it.
Any other tips and tricks?
Serving video content is really not very different from serving any other type of content, it's just a large file. There aren't that many tips and tricks unique to static video.
What was the biggest bottleneck you had to deal with?
The speed at which they could horizontally scale when a big spike of traffic came in.
Getting that startup time down and dealing with a fan out in the API. Reducing the amount of requests that happen in the backend. They did this by reducing redundant calls.
How can you test with the Simian Army?
The Simian Army basically tests your ability to withstand different types of failure cases that happen in any distributed environment. Latency monkey is a big one because in a distributed environment, you have two types of outages: 1) not available and 2) slow.
Not available is fairly easy to detect. Slow is really hard to deal with.
First of all, what is slow? Well it depends on what you expect. So you have to figure that out before you can really tune everything.
Latency monkey really helps with testing this. It creates latency so you can see where your system breaks down. When that slows down, it's going to really affect your entire performance. Doing this in a controlled environment can dramatically help you fix these kinds of issues.
Can you think back to an issue that yo solved from using Chaos Monkey but really shocked you?
By the time he got there, Chaos Monkey had already been used and solved a lot of issues.
Chaos Gorilla did find some issues where data or requests were not distributed evenly and so when they lost a particular zone, the other two suddenly became far too overloaded to handle the issue. Most of these issues are now solved.
The interesting work is now happening with Chaos Kong. This is the one that shifts traffic from entire regions. While these issues aren't really affecting customers, there is room for improvement.
What are some things you automate that others don't think to?
A lot of people still aren't using Continuous Deployment, Continuous Integration, auto scaling, for example.
Why doesn't everyone use auto scaling?
Because it's hard. It's hard to get right. It's hard to tune your auto scaler, it's hard to figure out when it's working correctly.
Can you walk us through the process of deploying a change at Netflix?
You check in your code to the repository, which kicks off a bake, which in turn builds a full machine image that will run your code. That bake takes a few minutes.
That baked image gets launched in test, you go to run your tests, and then you launch it in production.
There is more automation being added so that the entire flow comes more naturally, but you still have to wait for it. So sometimes people will make changes while the machine is running in test and push the code out directly. If you're making application changes it works pretty well. Once it passes in test, they'll make a fully baked machine and launch it in production.
It's really up to each team to decide how they want to run their development.
Netflix maintains a culture of Freedom and Responsibility with its engineers. What does that mean? How is this different than the way other companies do it?
The idea is that you are a responsible adult and you have the freedom to do what you think is the right thing to do, and then you have to take responsibility for your actions.
The whole point is to get out of people's way.
For example, the security team doesn't approve anything. They monitor and scan and they suggest, but they are never a block to getting something done. They give suggestions and tools and the other engineers form best judgments from that.
What does Netflix use Cassandra for? Why Cassandra?
Cassandra is used as main storage for pretty much everything. There's an entire team of people that runs the Cassandra infrastructure.
I think that speaks volume for the software. As Jeremy points out, it also speaks volume for the team that maintains it!
What is CloudNative? What's the vision behind it?
The vision behind CloudNative is to eliminate the overhead of using Microservices. It essentially gives companies the ability to use the same architectural style that Netflix, Pinterest, and Airbnb all use, but without having to deal with the massive overhead.
They believe that it is the right way to build infrastructure for long-term reliability and scalability.
A lot of startups want to build it that way, but they don't because of the high cost. By providing this platform, startups can use it from the get-go.
It makes Microservices architecture simple.
Does this work only with baked AMIs or also with clean installs via Chef or Puppet, for example?
The system is designed for fully baked images, but you can use Chef, Puppet, Ansible, etc.. to create your image.
What are some best-practices you've learned to keep deployments as simple and as bullet-proof as possible - especially when deploying major infrastructure/data alterations? Asked by: willrstern
For data, the best thing is to have an intermediate step instead of stopping the database and changing the table. Instead, you bring up a new table that has the new schema, and you write a chunk of code that knows how to read and write to both the old and new tables, so that every time a request comes in it takes care of both tables. Then, you have a back channel process that migrates data that hasn't been migrated through that chunk of code from the old to the new table.
This technique gives you no downtime doing a schema change.
How do you get the skills necessary to work at the scale that you've worked at?
A lot of what Jeremy learned has been by just doing it. Find any opportunity to do something new and different, and taking that opportunity.
When Jeremy started his career, he was in IT and he would look for any opportunity where somebody needed a server setup or something new setup and he would say "yeah, I'll do it, I'll learn it, I'll spend the extra time to figure it out."
He ran a bunch of infrastructure at his house so that he would get a lot of experience that way.
The best way to learn this stuff is to just do it for something. If the company you are at doesn't let you do it for them, go and start your own project. Start a little hobby project-- you'll be surprised how much you can learn from it. That's really how he learned.
You have to go and seek out what you want to do.
When hiring SREs or any position, what do you look for? Technical expertise? Ambition?
For a Site Reliability Enginner specifically, Jeremy is looking for someone who has a passion for the customer and the customer experience.
At Netflix, for example, he wanted SREs who wanted to make sure that someone could always watch their movie.
As far as experience goes, someone who has experience with coding and running an infrastructure is going to be the best option. But, honestly, you can learn a skill. Obviously you are going to be better with experience, but the main skill you need to have is this ability to handle emergency situations and to stay cool under pressure. You also need to really be able to step back and think about the experience from the customer point of view. "How do we make it so the user gets their experience back as quickly as possible, and then, how do we diagnose the issue." That's really important.
A lot of people get stuck on figuring out what broke. So they'll start diving in and looking at logs. It's really important to understand that this is great, but more importantly, customers need to be back online first.
What are some scaling anti-patterns? (pertaining mostly to architecture - microservices, SOA, etc) Asked by: willrstern
A lot of people have a single database to do all of their joins but that is just going to get you into trouble because then you still have your single point of failure, it's just further back in the stack. Dividing up your database-- one database per service, basically. If you need to join data between them, then that gets moved down in the software. That's the really important part.
One pattern that people think is an anti-pattern is fully baking images.
Thanks for reading! Let us know if you have any questions in the comments below.
Alternatively, you can find Jeremy on twitter at @jedberg. You can also email him at jedberg at gmail.com.
How did this interview help you?
If you learned anything from this interview, please thank our guest for their time.
Oh, and help your followers learn by clicking the Tweet button below :)