Why you should use Processes instead of Threads to isolate loads in Python

Key Learning

Python uses a Global Interpreter Lock to make sure that  memory shared between threads isn’t corrupted. This is a design choice of the language that has it’s pros and cons. One of these cons is that in multi-threaded applications where at least one  thread applies a large load to the CPU, all other threads will slow down as well.

For multi-threaded Python applications that are at least somewhat time-sensitive, you should use Processes over Threads.

Experiment

I wrote a simple python script to show this phenomenon. Let’s take a look.

The core is this increment function. It takes in a Value and then sets it over and over, increment each loop, until the running_flag is set to false. The value of count_value is what is graphed later on, and is the measure of how fast things are going.

The other important bit is the load function:

Like incrementload is the target of a thread or process. The z variable quickly becomes large and computing the loop becomes difficult quickly.

The rest of the code is just a way to have different combinations of increment and load running at the same time for varying amounts of time.

Result

The graph really tells the story. Without the load thread running, the process and thread versions of increment run at essentially the same rate. When the load thread is running, increment  in a thread grinds to a halt compared to the process which is unaffected.

That’s all! I’ve pasted the full source below so you can try the experiment yourself.

Comparing blank string definition in Python3

In python3 using

or

Produces the same result for the programmer. Which one is faster? Using the python module timeit, it’s really easy to find out!

Using string="" is WAY faster.

Here’s the source code for my tests:

Creature Capture | Project Declaration & Top Level Flowchart

I’ve decided to embark on a video surveillance project! My family lives in a very rural part of the US, and constantly hear and see evidence of animals going crazy outside of my home at night. The goal of this project is to hopefully provide some kind of insight as to what animals actually live in my backyard.

Ideally, I want to monitor the yard using some kind if infrared motion detector. Upon a motion detection, an IR camera assisted by some IR spotlights would begin filming until it has been determined that there isn’t any more movement going on in yard. These clips would then be filed into a directory, and at the end of the night, they would be compiled and uploaded to YouTube. This video would then be sent to the user via email.

I’ve created the following flowchart to develop against as I begin implementing this idea.

I’ll be using a Raspberry Pi to implement this idea, a few months back I bought the IR camera module and haven’t used it for anything, this would be a good project to test it out.

There are a few hurtles that I’ll have to cross in order to make this project a success, like most groups of problems I deal with, they can be separated into hardware and software components.

Hardware

  1. Minimize false positives by strategically arranging motion detectors
  2. Make sure IR Spotlights are powerful enough to illuminate area
  3. Enclosure must be weatherproof & blend in with environment, Maine winters are brutal.

Software

  1. The Pi doesn’t have any built in software to take undetermined lengths of video.
  2. Must have a lot of error catching and other good OO concepts in order to ensure a long runtime.

I’ve actually come up with a routine for solving the first software problem I’ve listed, hopefully I’ll have an example of my solution in action later tonight.

Ideally, this project will have a working implementation completed by May 21, which is 7 days from now.

PiPlanter 2 | Python Modules & Text Overlays

So in my last posting of the PiPlanter source code, the python script alone was 500 lines long. The intent with was to make things more modular and generic compared to the original version of the code that ran two years ago. Since the project has expanded a considerable amount since two summers ago, my goal of keeping everything short and concise isn’t really valid anymore so I’ve decided to split the code up into modules.

This improves a number of things, but it makes it kind of inconvenient to simply paste the full version of the source into a blog post. To remedy this, I’ll be utilizing www.esologic.com/source, something I made years ago to host things like fritzing schematics.

The newest publicly available source version can be found here: http://192.168.1.37/source/PiPlanter_2/ along with some documentation and schematics for each version to make sure everything can get set up properly. What do you think of this change? Will you miss the code updates in the body text of a blog post?

With all that out of the way, let’s talk about the actual changes I’ve made since the last post.

The first and foremost is that using Pillow, I’ve added a way to overlay text onto the timelapse frames like so:

Before

After

 

This was prompted by some strange behavior by the plants I noticed recently seen here:

I thought it was strange how the chive seemed to wilt and then stand back up and then wilt again, it would have been nice to be able to see the conditions in the room to try and determine what caused this. Hopefully I can catch some more behavior like this in the future.

Here is the new Image function with the text overly part included if you’re curious:

Now that I’ve got the PIL as part of this project, I’ll most likely start doing other manipulations / evaluations to the images in the future.

Okay! Thanks for reading.

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.

PiPlanter 2 | Adding Youtube Upload Functionality

In order to keep things moving quickly, I’ve decided to take a shortcut when it comes to uploading timelapse videos to youtube. I’ve decided to basically create a function that passes data to youtube-upload, a command line utility for linux that can upload videos very simply.

Here’s the function:

It should remind you a lot of “TryTweet” from the main version of the PiPlanter.

 

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.

PiPlanter 2 | Solving Broken Pipe Errors [Errno 32] in Tweepy

If I haven’t mentioned it already, https://twitter.com/piplanter_bot IS the new twitter account for PiPlanter. Like last time, I’m using the tweepy library for python to handle all things twitter for the project. What I’m NOT using this time is Flickr. From a design point of view, it wasn’t worth it. It was too complicated and had too many things that could go wrong for me to continue using it. Twitter is more than capable of hosting images, and tweepy has a very simple method of passing these images to twitter. Recently I moved the whole setup indoors and mounted it all onto a shelf seen here and it came with a set of strange problems.

Long story short, what I think happened was that since I moved them to a different location, the complexity of the images increased, causing an increase in the size of the images themselves. A broken pipe error implies that the entirety of the package sent to twitter wasn’t sent, causing the tweet not to go through. I first started to suspect this problem after seeing this:

 

The graphs were going through just fine, but images were seeming to have a hard time. You can’t tell from this photo, but those tweets are hours apart as opposed to the 20 minutes they are supposed to be. Once I started having this problem, I bit the bullet and integrated logging into my project which produced this log:

Hours and hours of failed tweets due to “[Errno 32] Broken pipe”. I tried a lot of things, I figured out that it was the size of the images after seeing this:

Photos that were simple in nature had no problem being sent. After scaling the image size down, I’ve had absolutely no problem sending tweets.


If you are tweeting images with tweepy in python and getting intermediate Broken pipe errors, decrease the size of your image.
Thanks for reading.

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.

PiPlanter 2 | Progress Update

I’m almost done with a very stable version of the Python code running the PiPlanter. There are many specific differences between this version of the python code and the version I wrote and implemented last summer, but the main one is that I tried to write functions for pretty much every task I wanted to do, and made each routine much more modular instead of one long line after line block to do each day. This took significantly longer to do (thus the lack of updates, sorry) but is much more expandable going forward. Below is the new version of the code, but by no means am I an expert programmer. The following code seems to work very well for what I want it to do.

Note the distinct lack of comments. I will put out a much more polished version of the code when it’s done. Before I move onto things like a web UI etc, I would like to do a few more things with this standalone version. The above version renders videos into time lapses, I would like to be able to upload those videos somewhere, hopefully youtube. I would also like to be able to email the log file to the user daily, which should be easier than uploading videos to youtube.

The script that renders the MySQL data into a graph is the following, it on the other hand has not changed much at all since last year and is still the best method to render graphs like I want to:

Here are some photos of the current setup, it hasn’t changed much since last time:

Thank you very much for reading.

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.

Server Upgrade | Stress Testing

So I wrote a program that makes really big numbers in python in an attempt to break a VM. Here’s the code:

I left it on for a few days and ended up with this:

Someone try and beat 26 compounds!

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.

DSFU – Adding Email Functionality, Better User Experience, Stable Set Adding

Big post for this project, here’s a video:

This version of the code implements a few really cool features.

First things first I added 10 LEDs that display the percent uploaded of the batch. For example if 13 / 100 photos have been uploaded, the first LED will light up. If 56 / 100 the first 5 LEDs will light up. Eventually the 10 junk LEDs will be replaced with a bar graph which will be mounted externally on the front panel of the enclosure.

I am using every single available output on my Pi now, but I was able to get away with adding 1 more LED that I should be able to use by using a transistor array explained here:

On the code side of things, I updated the way photos are added to the set. It uses the same principal as described in the previous post (using APscheduler to do the adding on an interval). All of these changes can be seen below, it’s still very poorly commented however.

Thanks for reading!

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.

PiPlanter | Big Overhaul Update

Okay! So I leave for college in less than 30 days, but I’d like to make sure my tomatoes to continue to grow once I leave so I’ve taken some steps to make sure that my departure goes smoothly.

Here’s a video of my revised setup:

There are a few key differences between this setup and my previous one:

The main one is that the watering system has been 100% re-vamped. The water distribution happens via a hose with holes in it instead of using the tray at the bottom of the plant grid in the previous video.

It also takes, uploads and tweets a picture of itself using a raspberry pi camera module.

It also creates a new mysql table every two weeks, and in turn, renders a new kind of graph. The renderscript.php file receives an argument from the python script which is the table code.

Here’s the python script:

Here’s the .php script:

Thank you for reading!

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.