The Pipes V2 Engine: Limits and More

Here at Pipes, we’ve been working hard to prepare the V2 Engine for full deployment in August 2011. We realize that this is a huge change for most of you and your Pipes, so we are trying to make the transition as seamless as possible by giving you incremental updates and opportunities to test your Pipes with V2 functionality. Throughout this process, you’ve been giving us your feedback, and we’ve been listening. So, we’d like to take a moment to both reassure you about the V2 Pipes Engine and provide the previously-lacking information regarding limitations and how to best mitigate pain points in the migration process.

Many of you have been asking why V2? Why can’t we just keep the Pipes V1 Engine? Well, simply put, change is intrinsic to software. Products need to be maintained and updated on a regular basis to fix bugs, make use of improved technologies, and provide cool, new features to you — our awesome users. The Pipes V1 engine is nearly 4 years old and was originally built quickly to provide a proof-of-concept. No one could have predicted how hugely successful Pipes would become! And we are grateful to all our devoted users for years of commitment to the product. But now to prove our commitment to you, we are moving to the V2 Engine to provide an even greater Pipes product.

Now on the nitty gritty…

The V2 engine isn’t vastly “buggy” or “broken”. It’s just different. It uses the same engine as the YQL product. We wouldn’t be releasing V2 if we didn’t have high confidence in the engine. After comparing thousands of Pipes running in V1 versus V2, we’ve found only minor incompatibilities and have been working diligently to update V2 where it makes sense. But no process is perfect, and so we’ve been relying on you to report issues and findings.

There are however some explicit differences with the V2 Engine. As many of you have probably already noticed, we’ve imposed a few limitations on Pipes that previously did not exist in V1. Why? Primarily to reduce abuse, improve performance, and to scale. These limitations are in no way intended to prevent you from building feature-full Pipes. Rather, we just ask you to think differently in certain cases about how you create your Pipe. Anything that you were able to do with V1 you should be able to do in a slightly different way with V2.

Here’s an outline of the new V2 limitations in detail:

* 3 2 Subpipes (nesting level of 3): Pipe nesting using subpipes has been limited to a nesting depth of 3. For a single pipe, you can drag as many subpipe modules as you like on the editor. However, opening a subpipe and including another subpipe module can only be done up to a depth of 3. That is, we only allow pipe -> subpipe1 -> subpipe2. Subpipes are meant to be used just like functions in programming. If you keep nesting functions, the trace becomes so long and complex that it essentially defeats the purpose of modularizing functionality. With subpipes, nesting too deep actually negatively impacts the performance of your Pipe. This limit will affect both existing and new Pipes. If you get this error when running your pipe, you'll need to edit it and remove any/all subpipe modules at level 3 (the third tab in the editor).

* 10 Module Inputs: Many modules have dynamic inputs. Previously you could add as many inputs as you would like for feeds, parameters, etc. Now, we limit it to 10 fields per input parameter, with the exception of the URL Builder Module. This change is independent of the engine and was a feature we wanted to include with V1 but never got around to. Simply put, having too many input fields makes a module appear unruly. If you want to fetch many feeds for example, you can use multiple Fetch Feed modules and Union them together. Existing Pipes if left untouched will not be affected by this limit. However, if you begin editing a module with more than 10 inputs, the restriction will be enforced.

* 30sec Execution Timeout: All Pipes must run within a 30 second execution timeout. This is a restriction put in place by the underlying YQL engine. In the current implementation, we allot 27 seconds to fetching your external feeds and sources, and 3 seconds for the V2 engine to perform the Pipe processing on top of those feeds. Any feeds that take longer than 27 seconds will be cut off. Because the V2 engine is so powerful, 3 seconds is plenty of processing time. So if your V2 Pipes are hitting the timeout limit, chances are there are one ore more sources that are slow to fetch. In situations like those, you may want to take into consideration the quality of the source itself. For example, if a feed takes longer than 27 seconds to fetch, it’s likely a poor feed and should perhaps be replaced with a different source if possible.

* 1.5Mb Feed Size: YQL enforces a maximum response size of 1.5MB, so Pipes is subject to this as well. That is, we can currently only fetch 1.5Mb of data per feed. We will continue to investigate configuring this value and potentially adjusting it for Pipes.

So there you have it. We know it’s late in the game, but we hope we have helped clarify some core aspects of the Pipes V2 Engine. After the full migration to V2 in August, we will continue with regular, incremental updates to provide further improvements and features. If you have any more questions, concerns, encounter problems as you are upgrading your Pipes, or have feedback on the limitations, please let us know. All we ask is that you help maintain the integrity and relevance of this forum by only posting specific Pipes V2 related questions and issues. And as always, please include your pipe id so we can help you more quickly!

– The Pipes Team

Comments are closed.