How to schedule a Python Script on Google Cloud
Last updated
Last updated
Don’t have Airflow set up in your server? Don’t have any experience setting up a cron job to schedule your script? No worries. Today we are going to learn how to schedule your Python script in Google Cloud Platform with minimum knowledge of cron job.
Cloud Scheduler is a managed Google Cloud Platform (GCP) product that allows you specify a frequency in order to schedule a recurring job. In a nutshell, it is a lightweight managed task scheduler.
Why do we need a scheduler? A scheduler allows you to run a workflow at any timing. It retries in the event of failure and even lets you run something at 3 AM so that you don’t need to wake up in the middle of the night.
Let’s start with creating a Cloud Scheduler. Go to Google Cloud Platform to look for Cloud Scheduler or you can go to this link directly.
Note: You have to set up your billing account in order to use the Cloud Scheduler. You will 3 free jobs per month, per billing account. For more detail, you may refer to the Cloud Scheduler pricing.
Let’s get started with “Schedule a Job”.
In this example, I am going to schedule a Python script to send an email, so the job’s name will be schedule-email.
The advanced setting here is optional. But let’s set a retry attempts just to be safe. After that, let’s click the “Create” button to create the cloud scheduler.
Next, we have to set up the script with main.py and requirement.txt.
In this example, we are going to send a simple email with this scheduler. Hence, there is no additional package/library needed to this script. I am going to leave the requirement.txt unchanged.
The entry point will be the function you defined to run in the script, which is schedule_email in this example.
Yah! The email is scheduled successfully at 10 pm.
Now you have learned how to schedule a Python script with Google Cloud Scheduler. You can try this scheduler with other tasks like an ad hoc batch job, big data processing job, or infrastructure automation tooling. The nice part is that Cloud Scheduler handles all the heavy lifting for you!
You can view the main.py in my Github Gist. Cheers!
The schedule frequency is in unix-cron format. If you are not familiar with unx-cron, you may refer to the crontab guru website. I always refer to this website to make sure I set the right schedule.
For example, 30 8 * * * means this job will be running on a daily basis, at 08:30.
0 9 6 * * means this job will be running on a monthly basis, on every 6th at 09:00.
Next, we have to configure the job’s target. In this example, we will use Pub/Sub as the target type. A new topic schedule-emailis created a new topic for this job.
Next, let’s go to Cloud Function to create a cloud function.
Let’s configure our cloud function with name and region. The trigger type will be Pub/Sub and the topic will be the topic we just created which is schedule-email.
You also have the option to configure memory allocated, timeout and runtime environment variables. Kindly remember to allocate more time and memory if you are running a long script.
