Flink

Key Technologies # Flink Learn about how you can use Flink to solve a large number of problems in System Design. Many system design problems will require stream processing. You have a continuous flow of data and you want to process, transform, or analyze it in real-time. Stream processing is actually hard and expensive to get right. Many problems that seem like stream processing problems can actually be reduced to batch processing problems where you'd use something like Spark or (if you're ancient enough) Hadoop.Before embarking on a stream processing solution, ask yourself the critical question: "do I really need real-time latencies?". For many problems, the answer is no and the engineers after you will thank you for saving them the ops headache. The most basic example of this might be a service reading clicks from a Kafka topic, doing a trivial transformation (maybe reformatting the data for ingestion), and writing to a database. Easy. Simple Kafka Stream Processing But things can get substantially more complex from here. Imagine we want to keep track of the count of clicks per user in the last 5 minutes. Because of that 5 minute window, we've introduced *state* to our problem. Each message can't be processed independently because we need to remember the count from previous messages. While we can definitely do this in our service by just holding counters in memory, we've introduced a bunch of new problems. * As one example of a problem, if our new service crashes it will lose all of its state: basically the count for the preceding 5 minutes is gone. Our service could hypothetically recover from this by re-reading *all* the messages from the Kafka topic, but this is slow and expensive. * Or another problem is scaling. If we want to add a new service instance because we're handling more clicks, we need to figure out how to re-distribute the state from existing instances to the new ones. This is a complicated dance with a lot of failure scenarios! * Or what if events come in out of order or late! This is likely to happen and will impact the accuracy of our counts. And things only get *harder* from here as we add complexity and more statefulness. Fortunately, engineers have been building these systems for decades and have come up with useful abstractions. Enter one of the most powerful stream processing engines: **Apache Flink**. Flink is a framework for building stream processing applications that solves some of the tricky problems like those we've discussed above and more. While we could talk about Flink for days, in this deep dive we're going to focus on two different perspectives to understanding Flink: * First, we're going to talk about how Flink is used. There's a good chance you'll encounter a stream-oriented problem in your interview and Flink is a powerful, flexible tool for the job when it applies. * Secondly, you'll learn how Flink *works*, at a high-level, under the hood. Flink solves a lot of problems for you, but for interviews it's important you understand *how* it does that so you can answer deep-dive questions and support your design. We'll cover the important bits. Let's get to it! ## Basic Concepts ### Sources/Sinks ### Streams ### Operators ### State ### Watermarks ### Windows ## Basic Use ### Defining a Job ### Submitting a Job ### Sample Jobs Basic Dashboard Using RedisFraud Detection System ## How Flink Works ### Cluster Architecture Job Manager and Task ManagersTask Slots and Parallelism ### State Management State BackendsCheckpointing and Exactly-Once Processing ## In Your Interview ### Using Flink ### Lessons from Flink ## Conclusion ## References