Background
We recently read a blog post entitled “Lessons Learned from Running Apache Airflow at Scale” from Shopify. This post resonated with me for a couple of reasons, the first of which is that we immediately recognized the design because we delivered an almost-identical solution architecture to a client during the same timeframe. What impacted me the most though was my recollection of the interplay between Prominent Edge’s core values and many of the design decisions required by our project.
Our Client
Our client operates in a niche realm and is a smaller enterprise, with a limited Information Technology staff and budget. For this project, Prominent Edge would not only be designing and developing the solution, but also providing on-call DevSecOps and Engineering support. Because all of the work was greenfield, Prominent Edge also jointly developed the Product Roadmap with our client.
The Product
Our client envisioned an OSINT product with the potential need to scale massively, on relatively short notice, in response to tasking orders from our client’s customers. The product makes use of a number of Data Engineering pipelines and Machine Learning algorithms that were implemented to automate many of the tasks that were previously performed by analysts with specialized skills. The product performs targeted analysis of social media entities, news sites, and other public sources; for a predefined set of topics, with the ability to easily deploy targeted analysis of new ad hoc topics, typically in response to customer requests or world events.
Core Values Applied
Our client’s product vision was mature, with solid descriptions of many of the features that would be developed. While it is genuinely a pleasure to work with a client with such clearly defined goals, this also presented our first values-based dilemma.
Dilemma One: Competing Concerns
Prominent Edge is primarily a services company, providing professional engineering and consulting services to a variety of clients. Delivering this type of work is exactly what we exist to do. However, Prominent Edge is also (albeit on a smaller scale) a products company, so we are familiar with the risks and challenges associated with bringing a new product to market, particularly as a small business ourselves.
The dilemma presented by the conflict between short-term business gain versus several of our Core Values was plain. The specific values (copied in abridged form from our internal process documents) implicated in this scenario were:
We “do the right thing”. We execute our responsibilities in an open and accessible manner to the benefit of the team and the customers.
We empathize with our customers as well as others in the company by putting ourselves in their shoes to understand their point of view and also to proactively make decisions.
We resolved this dilemma by encouraging our client to strictly pursue a Minimum Viable Product (MVP), which would defer implementation of many of the features to a distant future. While this decision resulted in less work for Prominent Edge, it was absolutely the correct decision for our client and was the decision that best aligned with our Core Values. Plus, when customers come back to us for additional work or to take on additional projects, we will realize additional revenue. We want all our customers’ projects to be viable and sensible so that they can come back to us.
Dilemma Two: Choice of Technologies
Prominent Edge engineers had previous experience deploying solutions with several commonalities to the features envisioned by our client. As such, we felt that designing and deploying the system desired by our client would be relatively familiar and straightforward. The dilemma arose because the previous solutions were all Java-based architectures, while our client’s IT staff did not possess Java expertise.
While some companies might see this scenario as an opportunity to lock-in future work, such a decision would not align with our Core Values. In addition to the previously-described values of Integrity (particularly, doing what is “right” for our customer) and Empathy, we also value Service Orientation and Practicality.
We are committed to maximizing the return on investment for our customers by being efficient doing what is right for each customer.
We look for solutions that satisfy all the constraints of the particular problem.
In this case, our client did possess Python experience. With a bit of research and proof-of-concept work, our Architecture and Engineering staff felt confident that Apache Airflow represented a practical, Python-based solution that would satisfy our client’s needs and also be largely maintainable by our client’s own staff.
Dilemma Three: Portability
We felt very early in the engagement with our client that a cloud-based solution made overwhelming sense and our client was of the same opinion. While Prominent Edge has successfully deployed solutions on multiple different cloud providers including Amazon Web Services (AWS), Microsoft Azure, and Google Cloud (GCP); we deploy most frequently to AWS. Our client was unsure about committing to developing their new product on AWS, primarily because our client’s Data Scientist leading the effort had become interested in Azure’s Machine Learning service. This interest emerged from discussions between their Data Scientist and professional colleagues in other industries and our client wanted to leave-open the possibility of switching cloud providers in the future.
While it is generally easier (and some might even say “better”) to use a cloud provider’s services “out-of-the-box”, a trade-off exists in becoming tied too closely to that particular provider and sacrificing portability. We understood that this was a valid concern for our client and worked to define the degree of portability that would be acceptable. We jointly arrived at guidance of the form “reasonably portable, assuming there is a 50% chance you will need to port it within the next 18 months”. This resolution resulted from the practical application of our Core Value of Collaboration.
We foster a spirit of collaboration, and positive, integrated thinking among our staff and our customers. Customers are an extension of our team and we communicate accordingly.
In collaboration with our customer, we established portability between cloud providers as one of our key design constraints. For example, source code for Directed Acyclic Graphs (DAG’s) is versioned in Atlassian BitBucket, independent of any particular cloud provider and isolated from change in a future migration scenario. Clustered services are (Docker) containerized and orchestrated via Kubernetes, which provides ease of migration between cloud providers, among other benefits. While designing for portability did result in some more work, we are confident that the solution is readily portable to another cloud provider and satisfied our client’s goal.
The Solution
The product has been deployed for almost two years and, while growth has been steady and continuous, the solution currently operates at a scale an order-of-magnitude less than the solution described by Shopify. While Shopify deployed to Google Cloud, we deployed to AWS. Nevertheless, what Shopify has done further confirms that our solution is both scalable and portable.
Conclusion
The dilemmas that I shared are real and actually did occur as described. They represent just a few examples of the types of choices that we face in performing work for our customers. While our client is extremely satisfied with the work that we delivered on this project, we are most proud of the fact that it was delivered based on design decisions that aligned with our Core Values.
