) I've just found their offerings to be excellent. They have plenty of enterprise offerings, but everything included here is open source and there is no pay wall involved.Īnd no, I am not affiliated with Bitnami, although I have kids that eat a lot and don't have any particular ethical aversions to selling out. It would be a fairly large undertaking to do all this from scratch, so I use Bitnami. The configuration, environmental variables, and everything else acts the same. This means I can test and develop locally using my compose stack, build out new images, versions, packages, etc, and then deploy to Kubernetes. īitnami stacks (usually) work completely the same from their Docker Compose stacks to their Helm charts. In addition to popular community offerings, Bitnami, now part of VMware, provides IT organizations with an enterprise offering that is secure, compliant, continuously maintained and customizable to your organizational policies. In comes the Bitnami Apache Airflow docker compose stack for dev and Bitnami Apache Airflow Helm Chart for prod!īitnami makes it easy to get your favorite open source software up and running on any platform, including your laptop, Kubernetes and all the major clouds. Then, on top of the packages you need to configure database connections and a message queue. Also, even just installing those packages is a pain and I could rarely count on a rebuild actually working without some pain. I still use this approach for most of my other containers, including micro services that interact with my Airflow system, but configuring Airflow is a lot more than just installing packages. I used to roll my own Airflow containers using Conda. Generally, I either deploy the stack to production using either the same Docker compose stack if its a small enough instance that is isolated, or with Kubernetes when I need to interact with other services or file systems. Using Airflow allows me to concentrate on the business logic of what I'm trying to accomplish without getting too bogged down in implementation details.ĭuring that time I've adopted a set of systems that I use to quickly build out the main development stack with Docker and Docker Compose, using the Bitnami Apache Airflow stack. It could take place locally, on a Docker image, on Kubernetes, on any number of AWS services, on an HPC system, etc. My favorite feature of Airflow is how completely agnostic it is to the work you are doing or where that work is taking place. I've been using it for around 2 years now to build out custom workflow interfaces, like those used for Laboratory Information Management Systems (LIMs), Computer Vision pre and postprocessing pipelines, and to set and forget other genomics pipelines.
0 kommentar(er)
