Script Limitations

The xdotool and xte utilities are great for simple web page automation when the HTML form items are sequential and no special decision making is required. Unfortunately, I found that when I tried to book a park time on the weekend, I started to see some limitations (Figure 4). During busy times, if I tried to book by time, xte and xdotool could not determine whether the time slot was taken. A simple workaround would be to search for the first Available or Not Busy slot, but this doesn't allow you to pick times you like.

Figure 4: Limitations with keystroke automation.

For projects that require some logic (like choosing a good time from a list of times), Selenium with Python is an excellent fit.

Selenium with Python

Selenium [4] is a portable framework for testing web applications, with client-server tools and an IDE. The Selenium WebDriver component (available for Firefox, Google Chrome, Internet Explorer, Safari, Opera, and Edge) sends commands from client APIs directly to a browser. Client APIs are available for C#, Go, Java, JavaScript, PHP, Python, and Ruby. The Selenium Downloads page [5] has details on installation of the WebDriver scripts.

To install the Linux 32-bit Selenium driver (geckodriver) for Firefox, enter:

wget https://github.com/mozilla/geckodriver/releases/download/v0.29.1/geckodriver-v0.29.1-linux32.tar.gz
tar -xvzf geckodriver-v0.24.0-linux32.tar.gz
chmod +x geckodriver
sudo mv geckodriver /usr/local/bin

To install the Selenium library for Python, enter:

pip install selenium

The big difference between the xte or xdotool utility and Selenium is that Selenium can access the HTML code of the selected web page directly.

Log In with Selenium and Python

As for xte and xdotool, you need to do some background manual work before writing the script. Once the required web page is open, you can use the Web Developer Inspector tool to examine HTML code. To access the Inspector, Select Tools | Web Developer | Inspector from the top menubar or use the shortcut Ctrl+Shift+C.

For the Netflix sign-in example, the Email or phone number and Password inputs are needed (Figure 5). When the Inspector is open, items selected on the web page are highlighted in the Inspector pane. In this example, the Email or phone number entry uses id="id_userLoginId", and the password entry uses id="id_password". Listing 3 shows the Python code that signs in to Netflix.

01 #
02 # netflix_login.py - automate Netflix Login
03 #
04 from selenium import webdriver
05
06 url="https://www.netflix.com/ca/login"
07 email="my_email.com"
08 pwd="my_password"
09
10 browser = webdriver.Firefox()
11
12 browser.get(url)
13
14 # wait for page to refresh
15 browser.implicitly_wait(10)
16
17 username = browser.find_element_by_id('id_userLoginId')
18 username.send_keys(email)
19
20 password = browser.find_element_by_id('id_password')
21 password.send_keys(pwd)
22
23 password.submit()
24
25 print("Login Complete")

Figure 5: Finding input field IDs with Inspector.

When a web page is called, it's important to give the page some time to refresh. The implicity_wait(10) call (line 15) waits up to 10 seconds for a Selenium query to complete.

HTML items can be found by either ID (find_element_by_id()) or by name (find_element_by_name()). A Selenium object needs to be created before initiating any action on it. Line 17 finds and then creates a username object from ID 'id_userLoginId'. The send_keys() method is used to pass text strings to <input> tags (lines 18 and 21). Calling the submit() method on any input object will send all the form data as a request to the web server (line 23).