Not long ago, I found myself checking a website everyday or so to find out if a particular product became available (in stock). After a handful of times, I thought “why don’t I create a little script to do this for me, and forget about it in the meantime?”
Enter bash scripting + curl + grep
A few years ago, I would have used Selenium (or some other browser-automation tool) to solve this problem. After becoming more familiar with unix systems and all its handy tools, I now think bash is a much lighter weight alternative. A combination of bash, curl, and grep would do the trick.
The basic idea is to execute the
curl command as such:
curl -s http://www.foo.com | grep -q -i "bar"
What this does is execute a get request to
grep the response with the word bar. Then all we have to do is wrap it around an
if statement and perform some sort of action to let us know when the condition is met. In my case, I wanted to leverage Mac OS X’s notification center, so I found a way to trigger an AppleScript notification, which also shows up in Notification Center (with the added native Growl-like popup). The resulting snippet looks something like this:
if curl -s http://www.foo.com | grep -q -i "bar"; then osascript -e 'display notification "Found text in URL" with title "curl/grep test" sound name "default"' else echo "not found" fi
As you can tell, you can configure the message, title, and sound of the notification. Here’s a useful blogpost with more details on AppleScript notifications.
Now that we have the basic script set up, you may be wondering how to do this on some sort of loop so that we can have it triggered every so often. You may be inclined to writing a while loop of the like, but there’s a native, much better way to do this in unix systems.
crontab (short for cron table) is simply a command to manage shell commands to run on a set schedule. People usually refer to these as cron jobs. You can read more about Cron here.
As you can imagine, at this point we can simply (a) save our bash script as a *.sh file, and the create a cron job to execute it every certain time. With this, you can just go on with your life using your system regularly, and let this background task alert you when it finds what you want on whatever website.
Final shell script
I’ve put together a more generic form of this bash script in this github repo. It will allow you to customize it even further. You can download it and look at its usage (via
-h) for more details.