Deploying Apache Airflow through the Linode Marketplace

Apache Airflow is an open source workflow management platform for data engineering pipelines. You can use it to automate, orchestrate, and monitor workflows and data pipelines. One of Airflow’s greatest features is that you can create and execute workflows with code. When you use workflows that are powered by code, you can version control, collaborate on, and debug your workflows.

Deploying a Marketplace App

The Linode Marketplace lets you easily deploy software on a Compute Instance using Cloud Manager. See Get Started with Marketplace Apps for complete steps.

1. Log in to Cloud Manager and select the Marketplace link from the left navigation menu. This displays the Linode Create page with the Marketplace tab pre-selected.

2. Under the Select App section, select the app you would like to deploy.

3. Complete the form by following the steps and advice within the Creating a Compute Instance guide. Depending on the Marketplace App you selected, there may be additional configuration options available. See the Configuration Options section below for compatible distributions, recommended plans, and any additional configuration options available for this Marketplace App.

4. Click the Create Linode button. Once the Compute Instance has been provisioned and has fully powered on, wait for the software installation to complete. If the instance is powered off or restarted before this time, the software installation will likely fail.

To verify that the app has been fully installed, see Get Started with Marketplace Apps > Verify Installation. Once installed, follow the instructions within the Getting Started After Deployment section to access the application and start using it.

Configuration Options

• Supported distributions: Ubuntu 20.04 LTS
• Recommended plan: All plan types and sizes can be used.

Apache Airflow Options

• Email Address (required): The email address to use for generating SSL certificates.

Limited User (Optional)

You can optionally fill out the following fields to automatically create a limited user for your new Compute Instance. This is recommended for most deployments as an additional security measure. This account will be assigned to the sudo group, which provides elevated permission when running commands with the sudo prefix.

• Limited sudo user: Enter your preferred username for the limited user.
• Password for the limited user: Enter a strong password for the new user.
• SSH public key for the limited user: If you wish to login as the limited user through public key authentication (without entering a password), enter your public key here. See Creating an SSH Key Pair and Configuring Public Key Authentication on a Server for instructions on generating a key pair.
• Disable root access over SSH: To block the root user from logging in over SSH, select Yes (recommended). You can still switch to the root user once logged in and you can also log in as root through Lish.

Custom Domain (Optional)

If you wish to automatically configure a custom domain, you first need to configure your domain to use Linode’s name servers. This is typically accomplished directly through your registrar. See Use Linode’s Name Servers with Your Domain. Once that is finished, you can fill out the following fields for the Marketplace App:

• Linode API Token: If you wish to use the Linode’s DNS Manager to manage DNS records for your custom domain, create a Linode API Personal Access Token on your account with Read/Write access to Domains. If this is provided along with the subdomain and domain fields (outlined below), the installation attempts to create DNS records via the Linode API. See Get an API Access Token. If you do not provide this field, you need to manually configure your DNS records through your DNS provider and point them to the IP address of the new instance.
• Subdomain: The subdomain you wish to use, such as www for www.example.com.
• Domain: The domain name you wish to use, such as example.com.

Getting Started after Deployment

1. Log into your new Compute Instance through LISH or SSH using the root user and the password you entered when creating the instance.

2. If you log in on the same day you deployed the app, a message is displayed with a link to your Apache Airflow GUI and the username and password to use.

   The installation of Apache Airflow is now complete, and the application is running in standalone mode.
   #
   You can log into the Airflow GUI at 192-0-2-170.ip.linodeusercontent.com
   With the credentials:
   Username: admin
   Password: pNbgWNDhYcRNvgRq
   #
   Standalone mode is not recommended for production.

   If you don’t see this message, you can output it using the command below:

   cat /etc/motd

3. Open your web browser and navigate to the Airflow URL provided in the last step. This should be the custom domain you entered during deployment or your Compute Instance’s rDNS domain (such as 192-0-2-1.ip.linodeusercontent.com). See the Managing IP Addresses guide for information on viewing the rDNS.

4. Within the Airflow login prompt that appears, enter the credentials provided in a previous step and sign in.

5. Once you are signed in, the Airflow dashboard appears. From here, you can view the DAGs (Directed Acyclic Graphs) and access all other areas of the dashboard.

6. You can now start using Apache Airflow. If you are unfamiliar with it, consider reading through the official documentation or Linode’s own guides:

   • Airflow > Tutorials
   • Airflow > How-to Guides
   • Create Connections and Variables in Apache Airflow

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

• Apache Airflow