One plus One equals Zero

(disclaimer: this is a rant. no real insight should be expected)

Came across this well known mathematical/philosophical phrase by chance (sometimes: “one plus one equals one” or “one plus two equals zero”).

And this is what happened:

We came up with this product that provides a kind of ETL pipeline over data using sparks. This pipeline gives a simplified read-process-write for multiple data-sources using multiple processing scripts into any number of sink data targets.

For the initial POC release we based our work on an existing product, that is used to share data between organizations. This share product provided (being in its early version itself) only one share method, called “Copy-Based”.

Copy based looks like that:

1. I share a “pointer” to my data with you
2. When you want it, you create a “snapshot”
3. You ask for a copy of this snapshot into your own storage.
4. Now you have a copy, and you can sell it.

This method gives you an authentic copy of the data, and is simpler to secure on the backend. Disadvantages: It gives you stale data by definition, requires storage, and is pretty slow.

A more advanced option is “In-Place” method, that goes a bit like this:

1. I “allow” you to read my data.
2. You access the shared “pointer” and read my data “directly”
3. You are smarter now.

The second method saves storage cost, limits traffic cost to when you need it, and ensures the data you read is always up to date. It is also faster inherently, because you access the data as soon as it is available. Disadvantages: It is pretty hard to secure on the backend, and lose the inherent snapshot capability of the Copy-Based method.

When we leveraged this product, copy based method fit our needs perfectly for the early version, and this is how we used it:

1. The user gives us the simplified pipeline.
2. We identify all the data it consumes, and copy it into our processing environment.
3. We process away like there is no tomorrow.
4. We copy the results out into the users’ storage.

Then we said “Yes, we can do all that, but we are using Copy-Based method, and maybe some day, we’ll do the In-Place fandango.”

After a while we had some kind of architectural review with someone, and he looked at our product, and then he looked some more, and then he coughed and said:

“You are doing In-Place dude. As soon as the magik-pipeline is triggered, it pulls the data, processes it, and spits out a result. This is what In-Place is for!”

We were shocked and bewildered. It meant that we will have to rename all our Enums! But it also meant something else.

It was funny.

Our product provided In-Place processing that needs to create a snapshot from the source data, copy it for processing, need storage to save the result before you can access it, and by then it is already stale. Because it takes forever.

In other words, we managed to get all the disadvantages of Copy-Based and In-Place in one go!

Then I realized that in this case, One plus One equals Zero.