Please use the issue tracker for all comments and errata. This version covers the latest stable release of ZeroMQ 3. If you are using older versions of ZeroMQ then some of the examples and explanations won't be accurate. The Guide is originally in Cbut also in PHPPythonLuaand Haxe. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast.

You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. It has a score of language APIs and runs on most operating systems. ZeroMQ is from iMatix and is LGPLv3 open source. We took a normal TCP socket, injected it with a mix of radioactive isotopes stolen from a secret Soviet atomic research project, bombarded it with era cosmic rays, and put it into the hands of a drug-addled comic book author with a badly-disguised fetish for bulging muscles clad in spandex.

Yes, ZeroMQ sockets are the world-saving superheroes of the networking world. On the one hand this strange name lowers ZeroMQ's visibility on Google and Twitter. Originally the zero in ZeroMQ was meant as "zero broker" and as close to "zero latency" as possible. Since then, it has come to encompass different goals: More generally, "zero" refers to the culture of minimalism that permeates the project.

We add power by removing complexity rather than by exposing new functionality. This book is written for professional programmers who want to learn how to make the massively distributed software that will dominate the future of computing. We assume you can read C code, because most of the examples here are in C even though ZeroMQ is used in many languages. We assume you care about scale, because ZeroMQ solves that problem above all others. We assume you need the best possible results with the least possible cost, because otherwise you won't appreciate the trade-offs that ZeroMQ makes.

Other than that basic background, we try to present all the concepts in networking and distributed computing you will need to use ZeroMQ. Thanks to Andy Oram for making the O'Reilly book happen, and editing this text. Thanks to Bill Desmarais, Brian Dorsey, Daniel Lin, Eric Desgranges, Gonzalo Diethelm, Guido Goldstein, Hunter Ford, Kamil Shakirov, Martin Sustrik, Mike Castleman, Naveen Chawla, Nicola Peduzzi, Oliver Smith, Olivier Chamoux, Peter Alexander, Pierre Rouleau, Randy Dryburgh, John Unwin, Alex Thomas, Mihail Minkov, Jeremy Avnet, Michael Compton, Kamil Kisiel, Mark Kharitonov, Guillaume Aubert, Ian Barber, Mike Sheridan, Faruk Akgul, Oleg Sidorov, Lev Givon, Allister MacLeod, Alexander D'Archangel, Andreas Hoelzlwimmer, Han Holl, Robert G.

Jakabosky, Felipe Cruz, Marcus McCurdy, Mikhail Kulemin, Dr. Sherman, Eric Pugh, Tyler Sellon, John E. How to explain ZeroMQ? Some of us start by saying all the wonderful things it does. It's sockets on steroids. It's like mailboxes with routing.

Others try to share their moment of enlightenment, that zap-pow-kaboom satori paradigm-shift moment when it all became obvious. Things just become simpler. It opens the mind. Others try to explain by comparison. It's smaller, simpler, but still looks familiar. Personally, I like to remember why we made ZeroMQ at all, because that's most likely where you, the reader, still are today. Programming is science dressed up as art because most of us don't understand the physics of software and it's rarely, if ever, taught.

The physics of software is not algorithms, data structures, languages and abstractions. These are just tools we make, use, throw away. The real physics of software is the physics of people—specifically, our limitations when it comes to complexity, and our desire to work together to solve large problems in pieces. This is the science of programming: We live in a connected world, and modern software has to navigate this world.

So the building blocks for tomorrow's very largest solutions are connected and massively parallel. It's not enough for code to be "strong and silent" any more. Code has to talk to code. Code has to be chatty, sociable, well-connected. Code has to run like the human brain, trillions of individual neurons firing off messages to each other, a massively parallel network with no central control, no single point of failure, yet able to solve immensely difficult problems.

And it's no accident that the future of code looks like the human brain, because the endpoints of every network are, at some level, human brains. If you've done any work with threads, protocols, or networks, you'll realize this is pretty much impossible. Even connecting a few programs across a few sockets is plain nasty when you start to handle real life situations.

The cost would be unimaginable. Connecting computers is so difficult that software and services to do this is a multi-billion dollar business. So we live in a world where the wiring is years ahead of our ability to use it. We had a software crisis in the s, when leading software engineers like Fred Brooks believed there was no "Silver Bullet" to "promise even one order of magnitude of improvement in productivity, reliability, or simplicity".

Brooks missed free and open source software, which solved that crisis, enabling us to share knowledge efficiently. Today we face another software crisis, but it's one we don't talk about much. Only the largest, richest firms can afford to create connected applications.

There is a cloud, but it's proprietary. Our data and our knowledge is disappearing from our personal computers into clouds that we cannot access and with which we cannot compete. Who owns our social networks? It is like the mainframe-PC revolution in reverse. We can leave the political philosophy for another book. The point is that while the Internet offers the potential of massively connected code, the reality is that this is out of reach for most of us, and so large interesting problems in health, education, economics, transport, and so on remain unsolved because there is no way to connect the code, and thus no way to connect the brains that could work together to solve these problems.

There have been many attempts to solve the challenge of connected code. There are thousands of IETF specifications, each solving part of the puzzle. For application developers, HTTP is perhaps the one solution to have been simple enough to work, but it arguably makes the problem worse by encouraging developers and architects to think in terms of big servers and thin, stupid clients.

So today people are still connecting applications using raw UDP and TCP, proprietary protocols, HTTP, and Websockets.

It remains painful, slow, hard to scale, and essentially centralized. Distributed P2P architectures are mostly for play, not work. How many applications use Skype or Bittorrent to exchange data? Which brings us back to the science of programming. To fix the world, we needed to do two things.

One, to solve the general problem of "how to connect any code to any code, anywhere". Two, to wrap that up in the simplest possible building blocks that people could understand and use easily. We assume you are using at least version 3. We assume you are using a Linux box or something similar. We assume you can read C code, more or less, as that's the default language for the examples. The examples live in a public GitHub repository. The simplest way to get all the examples is to clone this repository:.

Next, browse the examples subdirectory. You'll find examples by language. If there are examples missing in a language you use, you're encouraged to submit a translation. This is how this text became so useful, thanks to the work of many people. So let's start with some code. We start of course with a Hello World example. We'll make a client and a server. The client sends "Hello" to the server, which replies with "World".

Here's the server in C, which opens a ZeroMQ socket on portreads requests on it, and replies with "World" to each request:. The REQ-REP socket pair is in lockstep. Doing any other sequence e. ZeroMQ uses C as its reference language and this is the main language we'll use for examples. If you're reading this online, the link below the example takes you to translations into other programming languages.

In a language like PHP or Java, we can hide even more and the code becomes even easier to read:. Now this looks too simple to be realistic, but ZeroMQ sockets have, as we already learned, superpowers.

You could throw thousands of clients at this server, all at once, and it would continue to work happily and quickly. For fun, try starting the client and then starting the server, see how it all still works, then think for a second what this means.

Let us explain briefly what these two programs are actually doing. They create a ZeroMQ context to work with, and a socket. Don't worry what the words mean. You'll pick it up. The server binds its REP reply socket to port The server waits for a request in a loop, and responds each time with a reply. The client sends a request and reads the reply back from the server. If you kill the server Ctrl-C and restart it, the client won't recover properly. Recovering from crashing processes isn't quite that easy.

Making a reliable request-reply flow is complex enough that we won't cover it until Chapter 4 - Reliable Request-Reply Patterns. There is a lot happening behind the scenes but what matters to us programmers is how short and sweet the code is, and how often it doesn't crash, even under a heavy load.

This is the request-reply pattern, probably the simplest way to use ZeroMQ. ZeroMQ doesn't know anything about the data you send except its size in bytes.

That means you are responsible for formatting it safely so that applications can read it back. Doing this for objects and complex data types is a job for specialized libraries like Protocol Buffers.

But even for strings, you need to take care. In C and some other languages, strings are terminated with a null byte. We could send a string like "HELLO" with that extra null byte:. However, if you send a string from another language, it probably will not include that null byte.

For example, when we send that same string in Python, we do this:. Then what goes onto the wire is a length one byte for shorter strings and the string contents as individual characters. And if you read this from a C program, you will get something that looks like a string, and might by accident act like a string if by luck the five bytes find themselves followed by an innocently lurking nullbut isn't a proper string.

When your client and server don't agree on the string format, you will get weird results. When you receive string data from ZeroMQ in C, you simply cannot trust that it's safely terminated. Every single time you read a string, you should allocate a new buffer with space for an extra byte, copy the string, and terminate it properly with a null.

So let's establish the rule that ZeroMQ strings are length-specified and are sent on the wire without a trailing null. In the simplest case and we'll do this in our examplesa ZeroMQ string maps neatly to a ZeroMQ message frame, which looks like the above figure—a length and some bytes. Here is what we need to do, in C, to receive a ZeroMQ string and deliver it to the application as a valid C string:. The result is zhelpers. It is a fairly long source, and only fun for C developers, so read it at leisure.

ZeroMQ does come in several versions and quite often, if you hit a problem, it'll be something that's been fixed in a later version. So it's a useful trick to know exactly what version of ZeroMQ you're actually linking with. The second classic pattern is one-way data distribution, in which a server pushes updates to a set of clients. Let's see an example that pushes out weather updates consisting of a zip code, temperature, and relative humidity. We'll generate random values, just like the real weather stations do.

Here is the client application, which listens to the stream of updates and grabs anything to do with a specified zip code, by default New York City because that's a great place to start any adventure:. If you don't set any subscription, you won't get any messages. It's a common mistake for beginners. The subscriber can set many subscriptions, which are added together. That is, if an update matches ANY subscription, the subscriber receives it.

The subscriber can also cancel specific subscriptions. A subscription is often, but not necessarily a printable string. The PUB-SUB socket pair is asynchronous. Trying to send a message to a SUB socket will cause an error. In theory with ZeroMQ sockets, it does not matter which end connects and which end binds.

However, in practice there are undocumented differences that I'll come to later. For now, bind the PUB and connect the SUB, unless your network design makes that impossible. There is one more important thing to know about PUB-SUB sockets: Even if you start a subscriber, wait a while, and then start the publisher, the subscriber will always miss the first messages that the publisher sends.

This is because as the subscriber connects to the publisher something that takes a small but non-zero timethe publisher may already be sending messages out. This "slow joiner" symptom hits enough people often enough that we're going to explain it in detail. Say you have two nodes doing this, in this order:.

Then the subscriber will most likely not receive anything. You'll blink, check that you set a correct filter and try again, and the subscriber will still not receive anything. Making a TCP connection involves to and from handshaking that takes several milliseconds depending on your network and the number of hops between peers.

In that time, ZeroMQ can send many messages. For sake of argument assume it takes 5 msecs to establish a connection, and that same link can handle 1M messages per second. During the 5 msecs that the subscriber is connecting to the publisher, it takes the publisher only 1 msec to send out those 1K messages. In Chapter 2 - Sockets and Patterns we'll explain how to synchronize a publisher and subscribers so that you don't start to publish data until the subscribers really are connected and ready.

There is a simple and stupid way to delay the publisher, which is to sleep. Don't do this in a real application, though, because it is extremely fragile as well as inelegant and slow. Use sleeps to prove to yourself what's happening, and then wait for Chapter 2 - Sockets and Patterns to see how to do this right.

The alternative to synchronization is to simply assume that the published data stream is infinite and has no start and no end. One also assumes that the subscriber doesn't care what transpired before it started up. This is how we built our weather client example. So the client subscribes to its chosen zip code and collects updates for that zip code. That means about ten million updates from the server, if zip codes are randomly distributed. You can start the client, and then the server, and the client will keep working.

You can stop and restart the server as often as you like, and the client will keep working. When the client has collected its hundred updates, it calculates the average, prints it, and exits.

This is how long it takes to receive and filter 10M messages on my laptop, which is an era Intel i5, decent but nothing special:. As a final example you are surely getting tired of juicy code and want to delve back into philological discussions about comparative abstractive normslet's do a little supercomputing.

Our supercomputing application is a fairly typical parallel processing model. In reality, workers run on superfast boxes, perhaps using GPUs graphic processing units to do the hard math. Here is the ventilator. It generates tasks, each a message telling the worker to sleep for some number of milliseconds:. Here is the worker application. It receives a message, sleeps for that number of seconds, and then signals that it's finished:. Here is the sink application.

It collects the tasks, then calculates how long the overall processing took, so we can confirm that the workers really were running in parallel if there are more than one of them:. The average cost of a batch is 5 seconds. When we start 1, 2, or 4 workers we get results like this from the sink:.

The pipeline pattern also exhibits the "slow joiner" syndrome, leading to accusations that PUSH sockets don't load balance properly.

If you are using PUSH and PULL, and one of your workers gets way more messages than the others, it's because that PULL socket has joined faster than the others, and grabs a lot of messages before the others manage to connect. If you want proper load balancing, you probably want to look at the load balancing pattern in Chapter 3 - Advanced Request-Reply Patterns. Having seen some examples, you must be eager to start using ZeroMQ in some apps. Before you start that, take a deep breath, chillax, and reflect on some basic advice that will save you much stress and confusion.

ZeroMQ applications always start by creating a contextand then using that for creating sockets. You should create and use exactly one context in your process. Technically, the context is the container for all sockets in a single process, and acts as the transport for inproc sockets, which are the fastest way to connect threads in one process.

If at runtime a process has two contexts, these are like separate ZeroMQ instances. If that's explicitly what you want, OK, but otherwise remember:.

In general, you want to do interesting ZeroMQ stuff in the children, and boring process management in the parent. Classy programmers share the same motto as classy hit men: When you use ZeroMQ in a language like Python, stuff gets automatically freed for you.

But when using C, you have to carefully free objects when you're finished with them or else you get memory leaks, unstable applications, and generally bad karma. Memory leaks are one thing, but ZeroMQ is quite finicky about how you exit an application.

The ZeroMQ objects we need to worry about are messages, sockets, and contexts. Luckily it's quite simple, at least in simple programs:. This is at least the case for C development. In a language with automatic object destruction, sockets and contexts will be destroyed as you leave the scope. If you use exceptions you'll have to do the clean-up in something like a "final" block, the same as for any resource. If you're doing multithreaded work, it gets rather more complex than this. We'll get to multithreading in the next chapter, but because some of you will, despite warnings, try to run before you can safely walk, below is the quick and dirty guide to making a clean exit in a multithreaded ZeroMQ application.

First, do not try to use the same socket from multiple threads. Please don't explain why you think this would be excellent fun, just please don't do it. Next, you need to shut down each socket that has ongoing requests. The proper way is to set a low LINGER value 1 secondand then close the socket.

If your language binding doesn't do this for you automatically when you destroy a context, I'd suggest sending a patch. Finally, destroy the context. This will cause any blocking receives or polls or sends in attached threads i. Catch that error, and then set linger on, and close sockets in that thread, and exit. Do not destroy the same context twice. It's complex and painful enough that any language binding author worth his or her salt will do this automatically and make the socket closing dance unnecessary.

Many applications these days consist of components that stretch across some kind of network, either a LAN or the Internet.

So many application developers end up doing some kind of messaging. Some developers use message queuing products, but most of the time they do it themselves, using TCP or UDP. These protocols are not hard to use, but there is a great difference between sending a few bytes from A to B, and doing messaging in any kind of reliable way. Let's look at the typical problems we face when we start to connect pieces using raw TCP.

Any reusable messaging layer would need to solve all or most of these:. I see it's efficient because it uses poll instead of select. But really, Zookeeper should be using a generic messaging layer and an explicitly documented wire level protocol. It is incredibly wasteful for teams to be building this particular wheel over and over. But how to make a reusable messaging layer? Why, when so many projects need this technology, are people still doing it the hard way by driving TCP sockets in their code, and solving the problems in that long list over and over?

It turns out that building reusable messaging systems is really difficult, which is why few FOSS projects ever tried, and why commercial messaging products are complex, expensive, inflexible, and brittle. IniMatix designed AMQP which started to give FOSS developers perhaps the first reusable recipe for a messaging system. AMQP works better than many other designs, but remains relatively complex, expensive, and brittle. It takes weeks to learn to use, and months to create stable architectures that don't crash when things get hairy.

Most messaging projects, like AMQP, that try to solve this long list of problems in a reusable way do so by inventing a new concept, the "broker", that does addressing, routing, and queuing. Brokers are an excellent thing in reducing the complexity of large networks. But adding broker-based messaging to a product like Zookeeper would make it worse, not better. It would mean adding an additional big box, and a new single point of failure.

A broker rapidly becomes a bottleneck and a new risk to manage. If the software supports it, we can add a second, third, and fourth broker and make some failover scheme. It creates more moving pieces, more complexity, and more things to break. And a broker-centric setup needs its own operations team. You literally need to watch the brokers day and night, and beat them with a stick when they start misbehaving.

You need boxes, and you need backup boxes, and you need people to manage those boxes. It is only worth doing for large applications with many moving pieces, built by several teams of people over several years. So small to medium application developers are trapped.

Either they avoid network programming and make monolithic applications that do not scale. Or they jump into network programming and make brittle, complex applications that are hard to maintain. Or they bet on a messaging product, and end up with scalable applications that depend on expensive, easily broken technology. There has been no really good choice, which is maybe why messaging is largely stuck in the last century and stirs strong emotions: What we need is something that does the job of messaging, but does it in such a simple and cheap way that it can work in any application, with close to zero cost.

It should be a library which you just link, without any other dependencies. No additional moving pieces, so no additional risk. It should run on any OS and work with any programming language. And this is ZeroMQ: Actually ZeroMQ does rather more than this. It has a subversive effect on how you develop network-capable applications. But message processing rapidly becomes the central loop, and your application soon breaks down into a set of message processing tasks.

It is elegant and natural. Two nodes in one process node is a threadtwo nodes on one box node is a processor two nodes on one network node is a box —it's all the same, with no application code changes. Let's see ZeroMQ's scalability in action. Here is a shell script that starts the weather server and then a bunch of clients in parallel:. As the clients run, we take a look at the active processes using the top command', and we see something like on a 4-core box:.

Let's think for a second about what is happening here. The weather server has a single socket, and yet here we have it sending data to five clients in parallel. We could have thousands of concurrent clients. The server application doesn't see them, doesn't talk to them directly.

So the ZeroMQ socket is acting like a little server, silently accepting client requests and shoving data out to them as fast as the network can handle it. And it's a multithreaded server, squeezing more juice out of your CPU. For applications that want to run on both v2. Traditional network programming is built on the general assumption that one socket talks to one connection, one peer. There are multicast protocols, but these are exotic.

We create threads of logic where each thread work with one socket, one peer. We place intelligence and state in these threads. In the ZeroMQ universe, sockets are doorways to fast little background communications engines that manage a whole set of connections automagically for you. You can't see, work with, open, close, or attach state to these connections. Whether you use blocking send or receive, or poll, all you can talk to is the socket, not the connections it manages for you. The connections are private and invisible, and this is the key to ZeroMQ's scalability.

This is because your code, talking to a socket, can then handle any number of connections across whatever network protocols are around, without change. A messaging pattern sitting in ZeroMQ scales more cheaply than a messaging pattern sitting in your application code. So the general assumption no longer applies. As you read the code examples, your brain will try to map them to what you know. You will read "socket" and think "ah, that represents a connection to another node".

You will read "thread" and your brain will again think, "ah, a thread represents a connection to another node", and again your brain will be wrong.

If you're reading this Guide for the first time, realize that until you actually write ZeroMQ code for a day or two and maybe three or four daysyou may feel confused, especially by how simple ZeroMQ makes things for you, and you may try to impose that general assumption on ZeroMQ, and it won't work. And then you will experience your moment of enlightenment and trust, that zap-pow-kaboom satori paradigm-shift moment when it all becomes clear.

In Chapter 1 - Basics we took ZeroMQ for a drive, with some basic examples of the main ZeroMQ patterns: In this chapter, we're going to get our hands dirty and start to learn how to use these tools in real programs. To be perfectly honest, ZeroMQ does a kind of switch-and-bait on you, for which we don't apologize.

It's for your own good and it hurts us more than it hurts you. ZeroMQ presents a familiar socket-based API, which requires great effort for us to hide a bunch of message-processing engines.

However, the result will slowly fix your world view about how to design and write distributed software.

Sockets are the de facto standard API for network programming, as well as being useful for stopping your eyes from falling onto your cheeks. One thing that makes ZeroMQ especially tasty to developers is that it uses sockets and messages instead of some other arbitrary set of concepts.

Kudos to Martin Sustrik for pulling this off. It turns "Message Oriented Middleware", a phrase guaranteed to send the whole room off to Catatonia, into "Extra Spicy Sockets!

Like a favorite dish, ZeroMQ sockets are easy to digest. Sockets have a life in four parts, just like BSD sockets:. Note that sockets are always void pointers, and messages which we'll come to very soon are structures. As a mnemonic, realize that "in ZeroMQ, all your sockets are belong to us", but messages are things you actually own in your code.

Creating, destroying, and configuring sockets works as you'd expect for any object. But remember that ZeroMQ is an asynchronous, elastic fabric. This has some impact on how we plug sockets into the network topology and how we use the sockets after that. Thus we say that we "bind a socket to an endpoint" and "connect a socket to an endpoint", the endpoint being that well-known network address.

ZeroMQ connections are somewhat different from classic TCP connections. The main notable differences are:. There are sometimes issues of addressing: It also depends on the kind of sockets you're using, with some exceptions for unusual network architectures.

We'll look at socket types later. Now, imagine we start the client before we start the server. In traditional networking, we get a big red Fail flag. But ZeroMQ lets us start and stop pieces arbitrarily. A server node can bind to many endpoints that is, a combination of protocol and address and it can do this using a single socket. This means it will accept connections across different transports:. With most transports, you cannot bind to the same endpoint twice, unlike for example in UDP.

The ipc transport does, however, let one process bind to an endpoint already used by a first process. It's meant to allow a process to recover after a crash. Although ZeroMQ tries to be neutral about which side binds and which side connects, there are differences. We'll see these in more detail later. The upshot is that you should usually think in terms of "servers" as static parts of your topology that bind to more or less fixed endpoints, and "clients" as dynamic parts that come and go and connect to these endpoints.

Then, design your application around this model. The chances that it will "just work" are much better like that. The socket type defines the semantics of the socket, its policies for routing messages inwards and outwards, queuing, etc.

You can connect certain types of socket together, e. Sockets work together in "messaging patterns". We'll look at this in more detail later. It's the ability to connect sockets in these different ways that gives ZeroMQ its basic power as a message queuing system. There are layers on top of this, such as proxies, which we'll get to later. But essentially, with ZeroMQ you define your network architecture by plugging pieces together like a child's construction toy.

Let's look at the main differences between TCP sockets and ZeroMQ sockets when it comes to working with data:. It does not block except in some exception cases.

ZeroMQ provides a set of unicast transports inprocipcand tcp and multicast transports epgm, pgm. Multicast is an advanced technique that we'll come to later. Don't even start using it unless you know that your fan-out ratios will make 1-to-N unicast impossible.

For most common cases, use tcpwhich is a disconnected TCP transport. It is elastic, portable, and fast enough for most cases. We call this disconnected because ZeroMQ's tcp transport doesn't require that the endpoint exists before you connect to it.

Clients and servers can connect and bind at any time, can go and come back, and it remains transparent to applications. The inter-process ipc transport is disconnected, like tcp. It has one limitation: By convention we use endpoint names with an ". On UNIX systems, if you use ipc endpoints you need to create these with appropriate permissions otherwise they may not be shareable between processes running under different user IDs.

You must also make sure all processes can access the files, e. The inter-thread transport, inprocis a connected signaling transport. It is much faster than tcp or ipc. This transport has a specific limitation compared to tcp and ipc: This is something future versions of ZeroMQ may fix, but at present this defines how you use inproc sockets.

We create and bind one socket and start the child threads, which create and connect the other sockets. A common question that newcomers sbi mutual fund - sbi gold exchange traded scheme - growth option ZeroMQ ask it's one I've asked myself is, "how do I write an XYZ server in ZeroMQ?

The answer used to be "this is not how it works". ZeroMQ is not a neutral carrier: This framing is not compatible with existing protocols, which tend to use their own framing. The HTTP request uses CR-LF as its simplest framing delimiter, whereas ZeroMQ uses a length-specified frame. So you could write an HTTP-like protocol using ZeroMQ, using for example the request-reply socket pattern. But it would not be HTTP.

You could use this to read and write proper HTTP requests and responses. Hardeep Singh contributed this change so that he could connect to Telnet servers from his ZeroMQ application. At time of writing this is still somewhat experimental, but it shows how ZeroMQ keeps evolving to solve new problems. Maybe the next patch will be yours. We've seen that one socket can handle dozens, even thousands of connections at once.

This has a fundamental impact on how you write applications. A traditional networked application has one process or one thread per remote connection, and that process or thread handles one socket.

ZeroMQ lets you collapse this entire structure into a single process and then break it up as necessary for scaling. If you are using ZeroMQ for inter-thread communications only i. It's not a significant optimization though, more of a curiosity. Underneath the brown paper wrapping of ZeroMQ's socket API lies the world of messaging patterns. If you have a background in enterprise messaging, or know UDP well, these will be vaguely familiar.

But to most ZeroMQ newcomers, they are a surprise. We're so used to the TCP paradigm where a socket maps one-to-one to another node.

Let's recap briefly what ZeroMQ does for you. It delivers blobs of data messages to nodes, quickly and efficiently. You can map nodes to threads, processes, or nodes. ZeroMQ gives your applications a single socket API to work with, no matter what the actual transport like in-process, inter-process, TCP, or multicast.

It automatically reconnects to peers as they come and go. It queues messages at both sender and receiver, as needed. It limits these queues to guard processes against running out of memory. It handles socket errors. It uses lock-free techniques for talking between nodes, so there are never locks, waits, semaphores, or deadlocks. But cutting through that, it routes and queues messages according to precise recipes called patterns. It is these patterns that provide ZeroMQ's intelligence.

They encapsulate our hard-earned experience of the best ways to distribute data and work. ZeroMQ's patterns are hard-coded but future versions may allow user-definable patterns. ZeroMQ patterns are implemented by pairs of sockets with matching types.

In other words, to understand ZeroMQ patterns you need to understand socket types and how they work together. Mostly, this just takes study; there is little that is obvious at this level. We looked at the first three of these in Chapter 1 - Basicsand we'll see the exclusive pair pattern later in this chapter.

These are the socket combinations that are valid for a connect-bind pair either side can bind:. You'll also see references to XPUB and XSUB sockets, which we'll come to later they're like raw versions of PUB and SUB. Any other combination will produce undocumented and unreliable results, and future versions of ZeroMQ will probably return errors if you try them.

You can and will, of course, bridge other socket types via code, i. These four how much does it cost to buy stock in mcdonalds patterns are binary option strategies zero hour into ZeroMQ.

On top of those, we add high-level messaging patterns. We build these high-level patterns on top of ZeroMQ and implement them in whatever language we're using for our application. They are not part of the core library, do not come with the ZeroMQ package, and exist in their own space as part of the ZeroMQ community. For example the Majordomo pattern, which we explore in Chapter 4 - Reliable Request-Reply Patternssits in the GitHub Majordomo project in the ZeroMQ organization.

One of the things we aim to provide you with in this australian stock market current prices are a set of such high-level patterns, both small how to handle messages sanely and chapter 13 bankruptcy stock options how to make a reliable pub-sub architecture.

The libzmq core library has in fact two APIs to send and receive messages. On the wire, ZeroMQ messages are blobs of any size from zero upwards that fit ludacris - money maker mp3 free memory.

You do your own serialization using protocol buffers, msgpack, JSON, or whatever else your applications need to speak. It's wise to choose a data representation that is portable, but you can make your own decisions about trade-offs.

Here are the basic ground rules for using ZeroMQ messages in C:. This does not copy the data but copies a reference. You can then send the message twice or more, if you create more copies and the message will only be finally destroyed when the last copy is sent or closed. ZeroMQ also supports multipart messages, which let you send or receive a list of frames as a single on-the-wire message.

This is widely used in big movers stock market applications how to make money for christmas gift tags with cricut we'll look at that later in this chapter and in Chapter 3 - Advanced Request-Reply Patterns.

Frames also called "message parts" in the ZeroMQ reference manual pages are the basic wire format for ZeroMQ making money playing online poker. A frame is a length-specified block of data.

The length can be zero upwards. If you've done any TCP programming you'll appreciate why frames are a buy used corral panels answer to the question "how much data am I supposed to read of this network socket now? There is a wire-level protocol called ZMTP that defines how ZeroMQ reads and writes frames on a TCP connection.

If you're interested in how this works, the spec is quite short. Originally, a ZeroMQ message was one frame, like UDP. We later extended this with multipart messages, which are quite simply series of frames with a "more" bit set to one, followed by one with that bit set to zero.

The ZeroMQ API then lets you write messages with a "more" flag and when you read messages, it lets you check if there's "more". In the low-level ZeroMQ API and the reference manual, therefore, there's some fuzziness about messages versus frames. So here's a useful lexicon:. This is a zero-copy method and is guaranteed to create trouble for you. Witkowski article fair trade marketing an alternative system for globalization and development are far more important things to learn about ZeroMQ before ivafe su stock options start to worry about shaving off microseconds.

This rich API can be tiresome to work with. The methods are optimized for performance, not simplicity. If you start using these you will almost definitely get them wrong until you've read the man pages with some care. So one of the main jobs of a good language binding is to wrap this API up in classes that are easier to use. What if we want to read from multiple endpoints at the same time?

The simplest way is to connect one socket to all the endpoints and get ZeroMQ to do the fan-in forex daily technicals us.

This is legal if the remote endpoints are in the same pattern, but it would be wrong to connect a PULL socket to a PUB endpoint. Let's start with a dirty hack, partly for the fun of not doing it right, but mainly because it lets me show you how to do nonblocking socket reads.

Here is a simple example of reading from two sockets using nonblocking reads. This rather confused program acts both as a subscriber to weather updates, and a worker for parallel tasks:. The cost of this approach is some additional latency on the first message the sleep at the end of the loop, when there are no waiting messages to process. This would be a problem in applications where submillisecond latency was vital. Also, you need to check the documentation for ford market quotes stock premarket trading premarket or whatever function you use to make sure it does not busy-loop.

You can treat the sockets fairly by reading first from one, then the second rather than prioritizing them as we did in this example. ZeroMQ lets us compose a message out of several frames, giving us a "multipart message". Realistic applications use multipart messages heavily, both for wrapping messages with address information and for simple serialization. We'll look at reply envelopes later. What we'll learn now is simply how to blindly and safely read and write multipart messages in any application such as a proxy that needs to forward messages without inspecting them.

Here is how we send the frames in big movers stock market multipart message we receive each frame into a message object:. ZeroMQ aims for decentralized intelligence, but that doesn't mean your network is empty space in the middle.

It's filled with message-aware infrastructure and quite often, we build that infrastructure with ZeroMQ. The ZeroMQ plumbing can range from tiny pipes to full-blown service-oriented brokers. The messaging industry calls this intermediationmeaning that the stuff in the middle deals with either side.

In ZeroMQ, we call these proxies, queues, forwarders, device, or brokers, depending on the context.

Apollo Guidance Computer - Wikipedia

This pattern is extremely common in the real world from option traders hedge fund is why our societies and economies are filled with intermediaries who have no other real function than to reduce the complexity and scaling costs of larger networks.

Real-world intermediaries are typically called wholesalers, distributors, managers, and so on. One of the problems you will hit as you design larger distributed architectures is discovery. That is, how do pieces know about each other?

It's especially difficult if pieces come and go, so we call this the "dynamic discovery problem". There are several solutions to dynamic discovery. The simplest is to entirely avoid it by hard-coding or configuring the network architecture so discovery is done by auto trade the forex. That is, when you add a new piece, you reconfigure the network to know about it. In practice, this leads to increasingly fragile and unwieldy architectures.

Let's say you have one publisher and a hundred subscribers. You connect each subscriber to the publisher by configuring a publisher endpoint in each subscriber. Subscribers are dynamic; the publisher is static.

Now say you add more publishers. Suddenly, it's not so easy any more. If you continue to connect each subscriber to each publisher, the cost of avoiding dynamic discovery gets higher and higher. There are quite a few answers to this, but the very simplest answer is to add an intermediary; that is, a static point in the network to which all other nodes connect. In classic messaging, this is the job of the message broker.

ZeroMQ doesn't come with a message devoluciones finanzas forex as such, but it lets us build intermediaries quite easily.

You might wonder, if all networks eventually get large enough to need intermediaries, why don't we simply have a message broker in place for all applications? For beginners, it's a fair compromise. Just always use a star topology, forget about performance, and things will usually work. However, message brokers are greedy things; in their role as central intermediaries, they become too complex, too stateful, and eventually a problem.

It's better to think of intermediaries as simple stateless message switches. A good analogy is an HTTP proxy; it's there, but doesn't have any special role. Adding a pub-sub proxy solves the dynamic discovery problem in our example. We set the proxy in the "middle" of the network. The earning money playing video games opens an XSUB socket, an XPUB socket, and binds earn money fast red dead redemption to well-known IP addresses and ports.

Then, all other processes connect to the proxy, instead of to each other. It becomes trivial to add more subscribers or publishers. We need XPUB and XSUB sockets because ZeroMQ does subscription forwarding from subscribers to publishers. XSUB and XPUB are exactly like SUB and PUB except they expose ford market quotes stock premarket trading premarket as special messages.

Clock recovery from random binary signals proxy has to forward these subscription messages from subscriber side to publisher side, by reading them from the XPUB socket and writing them to the XSUB socket. This is the main use case for XSUB and XPUB. However, in real cases we usually need to allow multiple services as well as multiple clients.

This lets us scale up the power of the service many threads or processes or nodes rather than just one. The only constraint is that services must be stateless, all state being in the request or in some shared storage such as a database. There are two ways to connect multiple clients to multiple servers. The brute force way is to connect each client socket to multiple service endpoints. One client socket can connect to multiple service sockets, and the REQ socket will then distribute requests among these services.

Let's say you connect a client socket to three service endpoints; A, B, and C. The client makes requests R1, R2, R3, R4. R1 and R4 go to service A, R2 goes to B, and R3 goes to service C. This design lets you add more clients cheaply.

You can also euro sterling exchange rate graph forecast more services. Each client will distribute its requests to the services. But each client has to know the service topology. If you have clients and then you decide to add three more services, you need to reconfigure and restart clients in order for the clients to know about the three new services.

That's clearly not the kind of thing we want to be doing at 3 a. Too many static pieces are like liquid concrete: What we want is something sitting in between clients and services that centralizes all knowledge of the topology. Ideally, we should be able to add course day forex once trading remove services or clients at any time without touching any other part of the topology.

So we'll write a little message queuing broker that gives us this flexibility. The broker binds to two endpoints, a frontend for clients and a backend for services.

It doesn't actually manage any queues explicitly—ZeroMQ does that all american trading cards dayton ohio on each socket. When you use REQ to talk to REP, you get a strictly synchronous request-reply dialog. The client sends a request. The service reads the request and sends a reply. The client then reads the reply. If either the client or the service try to do anything else e.

But our broker has to be nonblocking. Luckily, there are two sockets called DEALER and ROUTER that let you do nonblocking request-response. You'll see in Chapter 3 - Advanced Request-Reply Patterns how DEALER and ROUTER sockets let you build all kinds of asynchronous request-reply flows.

For now, biggest gainer stock market today just going to see how DEALER and ROUTER let us extend REQ-REP across an intermediary, that is, play n trade robbery woodstock ga little broker.

In this simple extended request-reply pattern, REQ talks to ROUTER and DEALER talks to REP. In between the DEALER and ROUTER, we have to have code like our broker that pulls messages off the one socket and shoves them onto the other.

The request-reply broker binds to two endpoints, one for clients to connect to the frontend socket and one for workers to connect to the backend. To test this broker, you will want to change your workers so they connect to the backend socket. Here is a client that shows what I mean:. The only static node is the broker in the middle. It turns out that the core loop in the previous section's rrbroker is very useful, and reusable.

It lets us build pub-sub forwarders and shared queues and other little intermediaries with very little effort. The two or three sockets, if we want to capture data must be properly connected, bound, and configured. If you're like most ZeroMQ users, at this stage your mind is starting to think, "What kind of evil stuff can I do if I plug random socket types into the proxy?

Buy admiral shares frequent request from ZeroMQ users is, "How do I connect my ZeroMQ network with technology X? The simple answer is to build a bridge. A protocol interpreter, if you like. A common bridging problem in ZeroMQ is to bridge two transports or networks.

As an example, czarina forex glorietta 1 going to write a little proxy that sits in between a publisher and a set of subscribers, bridging two networks.

The frontend socket SUB faces the internal network where the weather server is sitting, and the backend PUB faces subscribers on the external network. It subscribes to the weather service on the frontend socket, and republishes its data on the backend socket. It looks very similar to the earlier proxy example, but the key part is that the frontend and backend sockets are on two different networks.

We can use this model for example to connect a multicast network pgm transport to a tcp publisher. ZeroMQ's error handling philosophy is a mix of fail-fast and resilience.

Processes, we believe, should be as vulnerable as possible to internal errors, and as robust as possible against external attacks and errors. To give an analogy, a living cell will self-destruct if it detects a single internal error, yet it will resist attack from the outside by all means possible. Assertions, exchange rate indian rupee to eur pepper the ZeroMQ code, are absolutely vital to robust code; they forex traders anonymous have to how to earn money from internet wikihow on the right side of the cellular wall.

And there should be such a wall. If it is unclear whether a fault is internal or external, that is a design flaw to be fixed. In other languages, you may get exceptions or halts.

When ZeroMQ detects yesterday's stock market external fault it returns an error to the calling code. In understanding the stock market and investing rare cases, it drops messages silently if there is no obvious strategy for recovering from the error.

In most of the C examples we've seen so far there's been no error handling. Real code should do error handling on every single ZeroMQ call. If you're using a language binding other than C, the binary option strategies zero hour may handle errors for you. In C, you do need to do this yourself.

There are some simple rules, starting with POSIX conventions:. It looks neat; then the optimizer removes all the asserts and the calls you want to make, and your application breaks in impressive ways. Let's see how to shut down a process cleanly. We'll take the parallel pipeline example from the previous section. If we've started a whole lot of workers in the background, we now want to kill them when the batch is finished.

Let's do this by sending a kill message to the workers. The best place to do this is the sink because it really knows when the batch is done. How do we connect the sink to the workers? We could switch to another socket type, or we could mix multiple socket flows. Let's try the latter: Here is the modified sink application. When it's finished collecting results, it broadcasts a kill message to all workers:.

Realistic applications need to shut down cleanly when interrupted with Ctrl-C or another signal such as SIGTERM. By default, these simply kill the process, meaning messages won't be flushed, files won't be closed cleanly, and so on. Thanks how much money does a six sigma black belt make your signal handler, your application will not die automatically.

Instead, you have a chance to clean up and exit gracefully. You have to now explicitly check for an interrupt and forex size positioning calculator it properly.

This sets up the signal handling. The interrupt will affect ZeroMQ calls as follows:. Any long-running application has to manage memory correctly, or eventually it'll use up all available memory and crash. If you use a language that handles this automatically for you, congratulations. ZeroMQ is perhaps the nicest way ever to write multithreaded MT applications.

Whereas ZeroMQ sockets require some readjustment if you are used to traditional sockets, ZeroMQ multithreading will take everything you know about writing MT applications, throw it into a heap in the garden, pour gasoline over it, and set it alight. It's a rare book that deserves burning, but most books on concurrent programming do.

Clock recovery from random binary signals - IEEE Xplore Document

To make utterly perfect MT programs and I mean that literallywe don't need mutexes, locks, or any other form of inter-thread communication except messages sent across ZeroMQ sockets.

By "perfect MT programs", I mean code that's easy to write and understand, that works with the same design approach in any icici direct trading holidays 2016 language, and on any operating system, and that scales across any number of CPUs with zero wait states and no point of diminishing returns.

If you've spent years learning tricks to make your MT code work at all, let alone rapidly, with locks and semaphores and critical sections, you will be disgusted when you realize it was all for nothing. It's like two drunkards trying to share a beer. It doesn't matter if they're good buddies.

Sooner or later, they're going to get into a fight. And the more drunkards you add to the table, the more they fight each other over the beer. The tragic majority of MT applications look like drunken bar fights.

The list of weird problems that you need to fight as you write classic shared-state MT code would be hilarious if it didn't translate directly into stress and risk, as code that seems to work suddenly fails under pressure.

A large firm with world-beating experience in buggy code released its list of "11 Likely Problems In Your Multithreaded Code", which covers forgotten synchronization, incorrect granularity, read and write tearing, lock-free reordering, lock convoys, two-step dance, and priority inversion. Yeah, we counted seven problems, not eleven. That's not the point though.

The point is, do you really want that code running the power grid or stock market to start getting two-step lock convoys at 3 p. Who cares what the terms actually mean? This is not what turned us on to programming, fighting ever more complex side effects with ever more complex hacks. Some widely used models, despite being the basis for entire industries, are fundamentally broken, and shared state concurrency is one of them. Code that wants crash affects stock market scale without limit does it like the Internet does, by sending messages and sharing nothing except a common contempt for broken programming models.

If you need to start more than one proxy in an application, for example, you will want to run each in their own thread. It is easy to make the emkay share stock brokers limited of creating the proxy frontend and backend sockets in one thread, and then passing the sockets to the proxy in another thread. This may appear to work at first but will fail randomly in real use.

Do not use or close sockets except in the thread that created them. If you follow these rules, you can quite easily build elegant multithreaded applications, and later split off threads into separate processes as you need to. Application logic can sit in threads, processes, or nodes: ZeroMQ uses native OS threads rather than virtual "green" threads. The advantage is that you don't need to learn any new threading API, and that ZeroMQ threads map cleanly to your operating system.

You can use standard tools like Intel's ThreadChecker to see what your application is doing. The disadvantages are that native threading APIs are not always portable, and that nikunj shah shares stock brokers ltd you have a huge number of threads in the thousandssome operating systems will get stressed. Let's see how this works in practice.

We'll turn our old Hello World server into something more capable. The original server ran in a single thread.

If the work per request is low, that's fine: But realistic servers have to do nontrivial work per request. A single core may not be enough when 10, clients hit the server all at once. So a realistic server will start multiple worker threads. It then accepts requests as fast as it can and distributes these to its worker threads. The worker threads grind through the work and eventually send their replies back. You can, of course, do all this using a proxy broker and external worker processes, but often it's easier to start one process that gobbles up sixteen cores than sixteen processes, each gobbling up one core.

Further, running workers as threads will cut out a how to make money selling shea butter hop, latency, and network traffic. The MT version of the Hello World service basically collapses the broker and workers into a single process:. Note that creating threads is not portable in most programming languages.

The POSIX library is pthreads, but on Windows you have to use a different API. We'll see in Chapter 3 - Advanced Request-Reply Patterns how to wrap this in a portable API.

Here the "work" is just a one-second pause. We could do anything in the workers, including talking to other nodes. Note how the request-reply chain is REQ-ROUTER-queue-DEALER-REP. When you start making multithreaded applications with ZeroMQ, you'll encounter the question of how to coordinate your threads.

Though you might be tempted to insert "sleep" statements, or use multithreading techniques such as semaphores or mutexes, the only mechanism that you should use are ZeroMQ messages. Remember the story of The Drunkards and The Beer Bottle. Let's make three threads that signal each other when they are ready. In this example, we use PAIR sockets over the inproc transport:. Note that multithreading code using this pattern is not scalable out to processes.

If you use inproc and socket pairs, you are building a tightly-bound application, i. Do this when low latency is really vital. The other design pattern is a loosely bound application, where threads have their own context and communicate over ipc or tcp.

You can easily break loosely bound threads into separate processes. This is the first time we've shown an example using PAIR sockets. Other socket combinations might seem to work, but they all have side effects that could interfere with signaling:.

When you want to coordinate a set of nodes on a network, PAIR sockets won't work well any more. This is one of the few areas stock exchange winners and losers the strategies for threads and nodes are different. Principally, nodes come and go whereas threads are usually static. PAIR sockets do not automatically reconnect if the remote node goes away and comes back.

The second significant difference between threads and nodes is that you typically have a fixed number of threads but a more variable number of nodes.

Let's take one of our earlier scenarios the weather server and clients and use node coordination to ensure that subscribers don't lose data how to make bux on tiny tower starting up.

In after market sks stocks case, we'll use a REQ-REP socket flow to synchronize subscribers and publisher.

Here is the publisher:. There are no guarantees that outbound connects will finish in any order whatsoever, if you're using any transport except inproc. ZeroMQ's message API lets you send and receive messages directly from and to application buffers without copying data.

We call this zero-copyand it can improve performance in some applications. You should think about using zero-copy in the specific case where you are sending large blocks of memory thousands of bytesat a high frequency.

For short messages, or for lower message rates, using zero-copy will make your code messier and more complex with no measurable benefit. Like all optimizations, use this when you know it helps, and measure before and after.

When you create the message, you also pass a function that ZeroMQ will call to free the how much money does a doctor of osteopathic medicine make of data, when it has finished sending the message. This is the simplest example, assuming buffer is a block of 1, bytes allocated on the heap:. There is no way to do zero-copy on receive: ZeroMQ delivers you a buffer that you can store as long as you wish, but it will not write data directly into application buffers.

On writing, ZeroMQ's multipart messages work nicely together with zero-copy. In traditional messaging, you need to marshal different buffers together into one buffer that you can send.

That means copying data. With ZeroMQ, you can send multiple buffers coming from different sources as individual message frames. Send each field as a length-delimited frame. To the application, it looks like a series of send and receive calls. But internally, the multiple parts get written to the network and read back with single system calls, so it's very efficient. In the pub-sub pattern, we can split the key into a separate message frame that we call an envelope. If you want to use pub-sub envelopes, make them yourself.

It's optional, and in previous pub-sub examples we didn't do this. Using a pub-sub envelope is a little more work for simple cases, but it's cleaner especially for real cases, where the key and the data are what a buy in karachi stock exchange separate things.

Subscriptions do a prefix match. That is, they look for "all messages starting with XYZ". The obvious question is: The best answer is to use an envelope because the match won't cross a frame boundary. Here is a minimalist example of how pub-sub envelopes look in code. This publisher sends messages of two types, A and B. This example shows that the subscription filter rejects or accepts the entire multipart message key plus data. You won't get part of a multipart message, ever.

If you subscribe to multiple publishers and you want to know their address so that you can send them data via another socket and this is a typical use casecreate a three-part message.

When you can send messages rapidly from process to process, you soon discover that memory is a precious resource, and one that can be trivially filled up. A few seconds of delay somewhere in a process can turn into a backlog that blows up a server unless you understand the problem and take precautions. The problem is this: Suddenly B gets very busy garbage collection, CPU overload, whateverand can't process the messages for a short period.

It could be a few seconds for some heavy garbage collection, or it could maimai asian market stockholm much longer, if there's a more serious problem.

What happens to the messages that process A is still trying to send frantically? Some will sit in B's network buffers. Some will sit on the Ethernet wire itself. Some will sit in A's network buffers. And the rest will accumulate in A's memory, as rapidly as the application behind A sends them. If you don't take some precaution, A can easily run out of memory and crash. It is a consistent, classic problem with message brokers. What makes it hurt more is that it's B's fault, superficially, and B is typically a user-written application which A has no control over.

What are the answers? One is to pass the problem upstream. A is getting the messages from somewhere else. So tell that process, "Stop! This is called flow control. It sounds plausible, but what if you're sending out a Twitter feed? Do you tell the whole world to stop tweeting while B gets its act together? Flow control works in some cases, but not in others. The transport layer can't tell the application layer to "stop" any more than a subway system can tell a large business, "please keep your staff at work for another half an hour.

The answer for messaging is to set limits on the size of buffers, and then when we reach those limits, to take some sensible action. In some cases not for a subway system, thoughthe answer is to throw away messages. In others, the best strategy is to wait.

ZeroMQ uses the concept of HWM high-water mark to define the capacity of its internal pipes. Some sockets PUB, PUSH only are incentive stock options subject to amt send buffers. Some SUB, PULL, REQ, REP only have receive buffers. Some DEALER, ROUTER, PAIR have both send and receive buffers. This was easy but also typically fatal for high-volume publishers.

If you're still using ZeroMQ v2. When your socket reaches its HWM, it will either block or drop data depending on the socket type. PUB and ROUTER sockets will drop data if they reach their HWM, while other socket types will block. Over the inproc transport, the sender and receiver share the same buffers, so the real HWM is the sum of the HWM set by both sides.

Lastly, the HWMs are not exact; while you may get up to 1, messages by default, the real buffer size may be much lower as little as halfdue to the way libzmq implements its queues.

As you build applications with ZeroMQ, you will come across this problem more than once: We have put together a diagram that walks through the most common causes for this.

In Chapter 2 - Sockets and Patterns we worked through the basics of using ZeroMQ by developing a series of small applications, each time exploring new aspects of ZeroMQ. We'll continue this approach in this chapter as we explore advanced patterns built on top of ZeroMQ's core request-reply pattern. We already looked briefly at multipart messages. Let's now look at a major use case, which is reply message envelopes.

An envelope is a way of safely packaging up data with an address, without touching the data itself. By separating reply addresses into an envelope we make it possible to write general purpose intermediaries such as APIs and proxies that create, read, and remove addresses no matter what the message payload or structure is. In the request-reply pattern, the envelope holds the return address for replies.

It is how a ZeroMQ network with no state can create round-trip request-reply dialogs. When you use REQ and REP sockets you don't even see envelopes; these sockets deal with them automatically.

But for most of the interesting request-reply patterns, you'll want to understand envelopes and particularly ROUTER sockets. We'll work through this step-by-step. A request-reply exchange consists of a request message, and an eventual reply message. In the simple request-reply pattern, there's one reply for each request. In more advanced patterns, requests and replies can flow asynchronously. However, the reply envelope always works the same way.

The ZeroMQ reply envelope formally consists of zero or more reply addresses, followed by an empty frame the envelope delimiterfollowed by the message body zero or more frames.

The envelope is created by multiple sockets working together in a chain. We'll break this down. We'll start by sending "Hello" through a REQ socket. The REQ socket creates the simplest possible reply envelope, which has no addresses, just an empty delimiter frame and the message frame containing the "Hello" string. This is a two-frame message. The REP socket does the matching work: Thus our original Hello World example used request-reply envelopes internally, but the application never saw them.

If you spy on the network data flowing between hwclient and hwserverthis is what you'll see: It doesn't seem to make much sense for a simple REQ-REP dialog. However you'll see the reason when we explore how ROUTER and DEALER handle envelopes. Now let's extend the REQ-REP pair with a ROUTER-DEALER proxy in the middle and see how this affects the reply envelope.

This is the extended request-reply pattern we already saw in Chapter 2 - Sockets and Patterns. We can, in fact, insert any number of proxy steps. The mechanics are the same. The ROUTER socket, unlike other sockets, tracks every connection it has, and tells the caller about these.

The way it tells the caller is to stick the connection identity in front of each message received.

An identity, sometimes called an addressis just a binary string with no meaning except "this is a unique handle to the connection". Then, when you send a message via a ROUTER socket, you first send an identity frame. Messages received are fair-queued from among all connected peers. As a historical note, ZeroMQ v2. There's some impact on network performance, but only when you use multiple proxy hops, which is rare. Mostly the change was to simplify building libzmq by removing the dependency on a UUID library.

Identities are a difficult concept to understand, but it's essential if you want to become a ZeroMQ expert. The ROUTER socket invents a random identity for each connection with which it works. If there are three REQ sockets connected to a ROUTER socket, it will invent three random identities, one for each REQ socket.

So if we continue our worked example, let's say the REQ socket has a 3-byte identity ABC. Internally, this means the ROUTER socket keeps a hash table where it can search for ABC and find the TCP connection for the REQ socket. The core of the proxy loop is "read from one socket, write to the other", so we literally send these three frames out on the DEALER socket. If you now sniffed the network traffic, you would see these three frames flying from the DEALER socket to the REP socket.

The REP socket does as before, strips off the whole envelope including the new reply address, and once again delivers the "Hello" to the caller. Incidentally the REP socket can only deal with one request-reply exchange at a time, which is why if you try to read multiple requests or send multiple replies without sticking to a strict recv-send cycle, it gives an error.

You should now be able to visualize the return path. When hwserver sends "World" back, the REP socket wraps that with the envelope it saved, and sends a three-frame reply message across the wire to the DEALER socket. Now the DEALER reads these three frames, and sends all three out via the ROUTER socket. The ROUTER takes the first frame for the message, which is the ABC identity, and looks up the connection for this.

If it finds that, it then pumps the next two frames out onto the wire. The REQ socket picks this message up, and checks that the first frame is the empty delimiter, which it is. The REQ socket discards that frame and passes "World" to the calling application, which prints it out to the amazement of the younger us looking at ZeroMQ for the first time. To be honest, the use cases for strict request-reply or extended request-reply are somewhat limited. For one thing, there's no easy way to recover from common failures like the server crashing due to buggy application code.

We'll see more about this in Chapter 4 - Reliable Request-Reply Patterns. However once you grasp the way these four sockets deal with envelopes, and how they talk to each other, you can do very useful things. We saw how ROUTER uses the reply envelope to decide which client REQ socket to route a reply back to. Now let's express this another way:. ROUTER sockets don't care about the whole envelope. They don't know anything about the empty delimiter. All they care about is that one identity frame that lets them figure out which connection to send a message to.

We have four request-reply sockets, each with a certain behavior. We've seen how they connect in simple and extended request-reply patterns. But these sockets are building blocks that you can use to solve many problems.

Here are some tips for remembering the semantics. DEALER is like an asynchronous REQ socket, and ROUTER is like an asynchronous REP socket. Where we use a REQ socket, we can use a DEALER; we just have to read and write the envelope ourselves.

Where we use a REP socket, we can stick a ROUTER; we just need to manage the identities ourselves. Think of REQ and DEALER sockets as "clients" and REP and ROUTER sockets as "servers". Mostly, you'll want to bind REP and ROUTER sockets, and connect REQ and DEALER sockets to them. It's not always going to be this simple, but it is a clean and memorable place to start. We've already covered a REQ client talking to a REP server but let's take one aspect: A REP server cannot talk to a REQ client that hasn't first sent it a request.

Technically, it's not even possible, and the API also returns an EFSM error if you try it. Now, let's replace the REQ client with a DEALER. This gives us an asynchronous client that can talk to multiple REP servers. If we rewrote the "Hello World" client using DEALER, we'd be able to send off any number of "Hello" requests without waiting for replies.

When we use a DEALER to talk to a REP socket, we must accurately emulate the envelope that the REQ socket would have sent, or the REP socket will discard the message as invalid. So, to send a message, we:. In the same way that we can replace REQ with DEALER, we can replace REP with ROUTER. This gives us an asynchronous server that can talk to multiple REQ clients at the same time.

If we rewrote the "Hello World" server using ROUTER, we'd be able to process any number of "Hello" requests in parallel. We saw this in the Chapter 2 - Sockets and Patterns mtserver example. In the first case, the ROUTER simply reads all frames, including the artificial identity frame, and passes them on blindly. In the second case the ROUTER must know the format of the reply envelope it's being sent. As the other peer is a REQ socket, the ROUTER gets the identity frame, an empty frame, and then the data frame.

Now we can switch out both REQ and REP with DEALER and ROUTER to get the most powerful socket combination, which is DEALER talking to ROUTER. It gives us asynchronous clients talking to asynchronous servers, where both sides have full control over the message formats. Because both DEALER and ROUTER can work with arbitrary message formats, if you hope to use these safely, you have to become a little bit of a protocol designer.

It depends on whether you actually need to send replies or not. You can swap a REP with a ROUTER, but you can also swap a REP with a DEALER, if the DEALER is talking to one and only one peer. When you replace a REP with a DEALER, your worker can suddenly go full asynchronous, sending any number of replies back. The cost is that you have to manage the reply envelopes yourself, and get them right, or nothing at all will work. We'll see a worked example later. Let's just say for now that DEALER to DEALER is one of the trickier patterns to get right, and happily it's rare that we need it.

This sounds perfect for N-to-N connections, but it's the most difficult combination to use. You should avoid it until you are well advanced with ZeroMQ. We'll see one example it in the Freelance pattern in Chapter 4 - Reliable Request-Reply Patternsand an alternative DEALER to ROUTER design for peer-to-peer work in Chapter 8 - A Framework for Distributed Computing. Mostly, trying to connect clients to clients, or servers to servers is a bad idea and won't work.

However, rather than give general vague warnings, I'll explain in detail:. The common thread in this valid versus invalid breakdown is that a ZeroMQ socket connection is always biased towards one peer that binds to an endpoint, and another that connects to that.

Further, that which side binds and which side connects is not arbitrary, but follows natural patterns. The side which we expect to "be there" binds: The side that "comes and goes" connects: Remembering this will help you design better ZeroMQ architectures. Let's look at ROUTER sockets a little closer. We've already seen how they work by routing individual messages to specific connections.

I'll explain in more detail how we identify those connections, and what a ROUTER socket does when it can't send a message. The identity concept in ZeroMQ refers specifically to ROUTER sockets and how they identify the connections they have to other sockets. More broadly, identities are used as addresses in the reply envelope. In most cases, the identity is arbitrary and local to the ROUTER socket: Independently, a peer can have an address that is physical a network endpoint like "tcp: An application that uses a ROUTER socket to talk to specific peers can convert a logical address to an identity if it has built the necessary hash table.

Because ROUTER sockets only announce the identity of a connection to a specific peer when that peer sends a message, you can only really reply to a message, not spontaneously talk to a peer.

This is true even if you flip the rules and make the ROUTER connect to the peer rather than wait for the peer to connect to the ROUTER. However you can force the ROUTER socket to use a logical address in place of its identity.

It works as follows:. Here is a simple example of two peers that connect to a ROUTER socket, one that imposes a logical address "PEER2":. ROUTER sockets do have a somewhat brutal way of dealing with messages they can't send anywhere: It's an attitude that makes sense in working code, but it makes debugging hard. The "send identity as first frame" approach is tricky enough that we often get this wrong when we're learning, and the ROUTER's stony silence when we mess up isn't very constructive.

Now let's look at some code. We'll see how to connect a ROUTER socket to a REQ socket, and then to a DEALER socket. These two examples follow the same logic, which is a load balancing pattern. This pattern is our first exposure to using the ROUTER socket for deliberate routing, rather than simply acting as a reply channel. The load balancing pattern is very common and we'll see it several times in this book.

It solves the main problem with simple round robin routing as PUSH and DEALER offer which is that round robin becomes inefficient if tasks do not all roughly take the same time.

It's the post office analogy. If you have one queue per counter, and you have some people buying stamps a fast, simple transactionand some people opening new accounts a very slow transactionthen you will find stamp buyers getting unfairly stuck in queues. Just as in a post office, if your messaging architecture is unfair, people will get annoyed. The solution in the post office is to create a single queue so that even if one or two counters get stuck with slow work, other counters will continue to serve clients on a first-come, first-serve basis.

One reason PUSH and DEALER use the simplistic approach is sheer performance. If you arrive in any major US airport, you'll find long queues of people waiting at immigration. The border patrol officials will send people in advance to queue up at each counter, rather than using a single queue.

Having people walk fifty yards in advance saves a minute or two per passenger. And because every passport check takes roughly the same time, it's more or less fair. This is the strategy for PUSH and DEALER: This is a recurring theme with ZeroMQ: The airport isn't the post office and one size fits no one, really well.

Let's return to the scenario of a worker DEALER or REQ connected to a broker ROUTER. The broker has to know when the worker is ready, and keep a list of workers so that it can take the least recently used worker each time.

The solution is really simple, in fact: The broker reads these messages one-by-one. Each time it reads a message, it is from the last used worker. And because we're using a ROUTER socket, we get an identity that we can then use to send a task back to the worker. It's a twist on request-reply because the task is sent with the reply, and any response for the task is sent as a new request.

The following code examples should make it clearer. Here is an example of the load balancing pattern using a ROUTER broker talking to a set of REQ workers:. The example runs for five seconds and then each worker prints how many tasks they handled. If the routing worked, we'd expect a fair distribution of work:. To talk to the workers in this example, we have to create a REQ-friendly envelope consisting of an identity plus an empty envelope delimiter frame. The synchronous versus asynchronous behavior has no effect on our example because we're doing strict request-reply.

It is more relevant when we address recovering from failures, which we'll come to in Chapter 4 - Reliable Request-Reply Patterns. The code is almost identical except that the worker uses a DEALER socket, and reads and writes that empty frame before the data frame. This is the approach I use when I want to keep compatibility with REQ workers. However, remember the reason for that empty delimiter frame: If we never need to pass the message along to a REP socket, we can simply drop the empty delimiter frame at both sides, which makes things simpler.

This is usually the design I use for pure DEALER to ROUTER protocols. The previous example is half-complete. It can manage a set of workers with dummy requests and replies, but it has no way to talk to clients. If we add a second frontend ROUTER socket that accepts client requests, and turn our example into a proxy that can switch messages from frontend to backend, we get a useful and reusable tiny load balancing message broker.

The difficult part of this program is a the envelopes that each socket reads and writes, and b the load balancing algorithm. We'll take these in turn, starting with the message envelope formats. Let's walk through a full request-reply chain from client to worker and back. In this code we set the identity of client and worker sockets to make it easier to trace the message frames. In reality, we'd allow the ROUTER sockets to invent identities for connections.

Let's assume the client's identity is "CLIENT" and the worker's identity is "WORKER". The client application sends a single frame containing "Hello". Because the REQ socket adds its empty delimiter frame and the ROUTER socket adds its connection identity, the proxy reads off the frontend ROUTER socket the client address, empty delimiter frame, and the data part. The broker sends this to the worker, prefixed by the address of the chosen worker, plus an additional empty part to keep the REQ at the other end happy.

This complex envelope stack gets chewed up first by the backend ROUTER socket, which removes the first frame. Then the REQ socket in the worker removes the empty part, and provides the rest to the worker application.

The worker has to save the envelope which is all the parts up to and including the empty message frame and then it can do what's needed with the data part. Note that a REP socket would do this automatically, but we're using the REQ-ROUTER pattern so that we can get proper load balancing. On the return path, the messages are the same as when they come in, i.

Now let's look at the load balancing algorithm. It requires that both clients and workers use REQ sockets, and that workers correctly store and replay the envelope on messages they get. You should now see that you can reuse and extend the load balancing algorithm with variations based on the information the worker provides in its initial "ready" message.

For example, workers might start up and do a performance self test, then tell the broker how fast they are. The broker can then choose the fastest available worker rather than the oldest.

We're going to push request-reply onto the stack and open a different area, which is the ZeroMQ API itself. There's a reason for this detour: Look at the core of the worker thread from our load balancing broker:. That code isn't even reusable because it can only handle one reply address in the envelope, and it already does some wrapping around the ZeroMQ API.

If we used the libzmq simple message API this is what we'd have to write:. And when code is too long to write quickly, it's also too long to understand.

Up until now, I've stuck to the native API because, as ZeroMQ users, we need to know that intimately. But when it gets in our way, we have to treat it as a problem to solve.

We can't of course just change the ZeroMQ API, which is a documented public contract on which thousands of people agree and depend. Instead, we construct a higher-level API on top based on our experience so far, and most specifically, our experience from writing more complex request-reply patterns.

What we want is an API that lets us receive and send an entire message in one shot, including the reply envelope with any number of reply addresses. One that lets us do what we want with the absolute least lines of code. Making a good message API is fairly difficult. We have a problem of terminology: ZeroMQ uses "message" to describe both multipart messages, and individual message frames.

We have a problem of expectations: And we have technical challenges, especially if we want to avoid copying data around too much. The challenge of making a good API affects all languages, though my specific use case is C. Whatever language you use, think about how you could contribute to your language binding to make it as good or better than the C binding I'm going to describe.

My solution is to use three fairly natural and obvious concepts: Here is the worker code, rewritten onto an API using these concepts:. Cutting the amount of code we need to read and write complex messages is great: Let's continue this process for other aspects of working with ZeroMQ. Here's a wish list of things I'd like in a higher-level API, based on my experience with ZeroMQ so far:. Turning this wish list into reality for the C language gives us CZMQa ZeroMQ language binding for C.

This high-level binding, in fact, developed out of earlier versions of the examples. It combines nicer semantics for working with ZeroMQ with some portability layers, and importantly for C, but less for other languages containers like hashes and lists. CZMQ also uses an elegant object model that leads to frankly lovely code.

One thing CZMQ provides is clean interrupt handling. This means that Ctrl-C will cause any blocking ZeroMQ call to exit with a return code -1 and errno set to EINTR. The high-level recv methods will return NULL in such cases. So, you can cleanly exit a loop like this:. So how about reactors? The CZMQ zloop reactor is simple but functional. It rebuilds its poll set each time you add or remove readers, and it calculates the poll timeout to match the next timer.

Then, it calls the reader and timer handlers for each socket and timer that need attention. The actual handling of messages sits inside dedicated functions or methods. You may not like the style—it's a matter of taste. What it does help with is mixing timers and socket activity. Getting applications to properly shut down when you send them Ctrl-C can be tricky. If you use the zctx class it'll automatically set up signal handling, but your code still has to cooperate.

If you have nested loops, it can be useful to make the outer ones conditional on!

uyanilalabiwi.web.fc2.com

If you're using child threads, they won't receive the interrupt. To tell them to shutdown, you can either:. In the ROUTER to DEALER example, we saw a 1-to-N use case where one server talks asynchronously to multiple workers. We can turn this upside down to get a very useful N-to-1 architecture where various clients talk to a single server, and do this asynchronously. The example runs in one process, with multiple threads simulating a real multiprocess architecture.

When you run the example, you'll see three clients each with a random IDprinting out the replies they get from the server. Look carefully and you'll see each client task gets 0 or more replies per request. Note that we're doing DEALER to ROUTER dialog between client and server, but internally between the server main thread and workers, we're doing DEALER to DEALER. If the workers were strictly synchronous, we'd use REP. However, because we want to send multiple replies, we need an async socket.

We do not want to route replies, they always go to the single server thread that sent us the request. Let's think about the routing envelope. The client sends a message consisting of a single frame. The server thread receives a two-frame message original message prefixed by client identity. We send these two frames on to the worker, which treats it as a normal reply envelope, returns that to us as a two frame message. We then use the first frame as an identity to route the second frame back to the client as a reply.

Now for the sockets: In this case, a DEALER to DEALER pattern is probably fine: Simplicity wins in this case. When you build servers that maintain stateful conversations with clients, you will run into a classic problem. If the server keeps some state per client, and clients keep coming and going, eventually it will run out of resources. Even if the same clients keep connecting, if you're using default identities, each connection will look like a new one.

We cheat in the above example by keeping state only for a very short time the time it takes a worker to process a request and then throwing away the state.

But that's not practical for many cases. To properly manage client state in a stateful asynchronous server, you have to:. Let's take everything we've seen so far, and scale things up to a real application.

We'll build this step-by-step over several iterations. Our best client calls us urgently and asks for a design of a large cloud computing facility. He has this vision of a cloud that spans many data centers, each a cluster of clients and workers, and that works together as a whole. Because we're smart enough to know that practice always beats theory, we propose to make a working simulation using ZeroMQ. Our client, eager to lock down the budget before his own boss changes his mind, and having read great things about ZeroMQ on Twitter, agrees.

Several espressos later, we want to jump into writing code, but a little voice tells us to get more details before making a sensational solution to entirely the wrong problem. So we do a little calculation and see that this will work nicely over plain TCP. It's a straightforward problem that requires no exotic hardware or protocols, just some clever routing algorithms and careful design. We start by designing one cluster one data center and then we figure out how to connect clusters together.

Workers and clients are synchronous. We want to use the load balancing pattern to route tasks to workers. Workers are all identical; our facility has no notion of different services. Workers are anonymous; clients never address them directly. We make no attempt here to provide guaranteed delivery, retry, and so on. For reasons we already examined, clients and workers won't speak to each other directly.

It makes it impossible to add or remove nodes dynamically. So our basic model consists of the request-reply message broker we saw earlier. Now we scale this out to more than one cluster. Each cluster has a set of clients and workers, and a broker that joins these together. There are a few possibilities, each with pros and cons:. Let's explore Idea 1. In this model, we have workers connecting to both brokers and accepting jobs from either one. However, it doesn't provide what we wanted, which was that clients get local workers if possible and remote workers only if it's better than waiting.

Also workers will signal "ready" to both brokers and can get two jobs at once, while other workers remain idle. It seems this design fails because again we're putting routing logic at the edges. So, idea 2 then. We interconnect the brokers and don't touch the clients or workers, which are REQs like we're used to.

This design is appealing because the problem is solved in one place, invisible to the rest of the world. Basically, brokers open secret channels to each other and whisper, like camel traders, "Hey, I've got some spare capacity. If you have too many clients, give me a shout and we'll deal". In effect it is just a more sophisticated routing algorithm: There are other things to like about this design, even before we play with real code:.

We'll now make a worked example. We'll pack an entire cluster into one process. That is obviously not realistic, but it makes it simple to simulate, and the simulation can accurately scale to real processes. This is the beauty of ZeroMQ—you can design at the micro-level and scale that up to the macro-level. Threads become processes, and then become boxes and the patterns and logic remain the same. Each of our "cluster" processes contains client threads, worker threads, and a broker thread.

There are several possible ways to interconnect brokers. What we want is to be able to tell other brokers, "we have capacity", and then receive multiple tasks. We also need to be able to tell other brokers, "stop, we're full". It doesn't need to be perfect; sometimes we may accept jobs we can't process immediately, then we'll do them as soon as possible.

The simplest interconnect is federationin which brokers simulate clients and workers for each other. We would do this by connecting our frontend to the other broker's backend socket. Note that it is legal to both bind a socket to an endpoint and connect it to other endpoints.

This would give us simple logic in both brokers and a reasonably good mechanism: The problem is also that it is too simple for this problem. A federated broker would be able to handle only one task at a time. If the broker emulates a lock-step client and worker, it is by definition also going to be lock-step, and if it has lots of available workers they won't be used.

Our brokers need to be connected in a fully asynchronous fashion. The federation model is perfect for other kinds of routing, especially service-oriented architectures SOAswhich route by service name and proximity rather than load balancing or round robin.

So don't dismiss it as useless, it's just not right for all use cases. Instead of federation, let's look at a peering approach in which brokers are explicitly aware of each other and talk over privileged channels. Let's break this down, assuming we want to interconnect N brokers. Each broker has N - 1 peers, and all brokers are using exactly the same code and logic.

There are two distinct flows of information between brokers:. Choosing good names is vital to keeping a multisocket juggling act reasonably coherent in our minds. Sockets do something and what they do should form the basis for their names.

It's about being able to read the code several weeks later on a cold Monday morning before coffee, and not feel any pain. Finding meaningful names that are all the same length means our code will align nicely. It's not a big thing, but attention to details helps. For each flow the broker has two sockets that we can orthogonally call the frontend and backend.

We've used these names quite often. A frontend receives information or tasks. A backend sends those out to other peers. The conceptual flow is from front to back with replies going in the opposite direction from back to front.