How to host private Python packages on Bitbucket for free, AND how to use them in a circleci pipeline

Bitbucket is great for hosting private git repos. Turns out, it can also be used to turn those repos into python packages that you can integrate into your projects with pip. This took a bit of trial and effort to make happen, let me know if there is anything additional you had to do to get things working on your end and I can add them to the guide.

Background

This whole process is built on pip’s ability to install packages from common VCS’s using SSH keys for access credentials. The syntax for doing that looks like this:

Pretty slick, you can even specify a branch or tag:

Since this repo is public, let’s try installing the package into a python virtual environment:

No dice. It didn’t work because our development environment isn’t configured correctly. Let’s get started with the guide.

Using private repo packages locally

Note: I’m on ubuntu 18.04, but I will leave Windows notes in each step if applicable.

Step 1: Make sure your repo CAN be installed as a python package

The key here is a proper setup.py file. Here are best the best set of docs I’ve found on how to make this file.
 
You can also look at the test repo for this project (https://bitbucket.org/esologic/sample_project/src/master/), it contains an example setup.py. This repo will also be the standard example for this post.
 
To make sure things are working correctly, you can try installing the package into your local python environment, or into a virtual one like I’m doing. Using sample_project as an example, we can do this like so:

If your package behaves as expected when installed like this locally, you’re all set to push the changes to your bitbucket repo and continue with the rest of the guide.

Step 2: Create SSH keys and add them to bitbucket

Note: at a few places in this step I use my own email as a reference, dev@esologic.com. Make sure whenever you see that, to substitute email address associated with your bitbucket account.
 
If you already have ssh keys created on your computer or whatever you’re developing on, they should be located at ~/.ssh. If you don’t see both id_rsa and id_rsa.pub files in that directory, create them with:

Leave passphrase blank.
 
Now, copy the contents of ~/.ssh/id_rsa.pub to bitbucket. The following images should walk you through the steps, make sure to give the key a memorable name.
Now, the ssh key of whatever dev environment you’re on is added to bitbucket.

Windows steps to create ssh keys

I followed these two (1, 2) guides to create ssh keys on windows.
The short version goes something like this:

Then follow the step above to add the keys to your bitbucket account.

Step 3: Make sure your account can read from the private repo with your python package

This is a simple, but a trap for young players. Make sure the account you’re trying to install the module with has at least read settings on the repo.

Since the Devon account is an owner of the repo, it will be allowed to read from the repo. The account ci_bot will also be able to read from the repo because it has read permissions.

Step 4: Install the package from bitbucket

With the bitbucket repo permissions set, and your SSH key added to your bitbucket account, you should be able to re-run the installation command from earlier and use the package.

Fantastic! Remember, your pip command git+ssh://git@bitbucket.org/esologic/sample_project.git will be different for your package. It will look something like this: git+ssh://git@bitbucket.org/{your username}/{your project}.git.

Any user that you give read permissions to on the repo will be able to install your package as well. This includes a machine user, so your CI builds can use your private package as well, which I’ll show you how to do next.

Using private repo packages in circleci

Bitbucket and circleci go together like peanut butter and chocolate. Adding CI to a bitbucket project is made fast and easy using circleci.

Step 5: Create a “machine user” in bitbucket

This user should have read only access to the package repo that you want to add to ci, so in this example it’s the sample_project repo.
 
You can accomplish this very easy through the BB ui, just make sure to keep track of whatever username and password you decide on.
 
Just to be clear, a machine user is just a regular bitbucket user that is only used by machines.

Step 6: Create SSH keys and add them to your machine user’s account

On whatever you system you have been using so far, enter the following commands and remember to leave passphrase blank.

Add the contents of ~/.ssh/ci_bot_keys/id_rsa.pub to bitbucket while signed in as your machine user like we did in step 2.

Step 7: Try git+ssh key insertion locally

(Note: you can skip this step, but if things don’t work when you add the step to your CI build start looking for errors here.)

By setting the environment variable GIT_SSH_COMMAND you can select which SSH key gets used by pip when doing an ssh pull.
 
Let’s try out the concept, and try out our new key locally. Run these two commands:

And then install your project like you did before. The package should install no problem, and you should see the same output as step 4.

Step 8: Set the $KEY environment variable in circleci

We now want to make the private key we made for our ci bot (~/.ssh/ci_bot_keys/id_rsa) available to the circle build process.
The only tricky part here is that the private key will contain newlines. For simplicity, we can replace them with underscores, and add the newlines back in the circle build.
 
Copy the output of this command to your clipboard:

The output ends after -----END RSA PRIVATE KEY-----_ in case your terminal doesn’t wrap correctly.
 
Now we need to set this value to the env var $KEY in the circleci build that we are trying to use our private package (sample_project) in.
 
Click the gear on the project page for your project in circle. For me, this brought me to https://circleci.com/bb/esologic/crossbow/edit, where crossbow is the name of my project.
 
Go to build settings -> Environment Variables and then set the variable like so:
 

Now that the variable is set, we need to change our circle config to use it.

Step 9: Add the step to your /.circleci/config.yml file

This does the same thing that we just tried locally, but in circle.
Make sure to put this step BEFORE the step that installs your python dependencies.

Make sure you select a circle image that has a git version of 2.17.0 or later, or this step will fail without an explanation. I found that the python image of circleci/python:3.7-buster worked when testing.

Try running your job, with this step added, it should be able to pull the package from your private repo. Let me know if you run into issues and I can try to help you out. Maybe donate the money you saved on hosting fees to me via paypal? 🤷💖

Thanks to

  • http://redgreenrepeat.com/2018/05/25/specifying-different-ssh-key-for-git/

Play multiple sound files on multiple output devices with Python and sounddevice

Ever wanted to have multiple different sound files playing on different output devices attached to a host computer? Say you’re writing a DJing application where you want one mix for headphones and one for the speakers. Or you’re doing some sort of kiosk or art installation where you have many sets of speakers that need to all be playing their own sound file but the whole thing needs to be synchronized. This would even be cool for something like an escape room.

The ladder example is where I needed this bit of code. I’ve been working with interdisciplinary artist Sara Dittrich on a few projects recently and she asked if I could come up with a way to play 8 different mono sound files on 8 different loudspeakers. Here’s a video of the whole setup in action, and an explanation of the project:

I’ve wrapped up all of the code for the art installation project, and that can be found in a github repo here. It includes the startup functionality etc. If you’re interested in recreating the video above, that repo would be a good starting place. The following is a list of the parts used to make that build happen:

Multi-Audio Example

It is worth it to give a simple example of how to play multiple files on multiple audio devices using python. I couldn’t find an examples on how to do this online and had to spend some time experimenting to make it all come together. Hopefully this saves you the trouble.

To install sounddevice on my Raspberry Pi, I had to run the following commands:

For this example, let’s say there are 4 audio files in the same directory as multi.py , so the directory looks like this:

The code is based on the sounddevice library for python, whose documentation is pretty sparse. This script will find the audio files, and then play them on as many devices as there are attached. For example, if you have 3 sound devices it will play 1.wav, 2.wav and 3.wav on devices 1-3. If you have any questions, feel free to ask:

Here are some more photos of the build:

Why you should use Processes instead of Threads to isolate loads in Python

Key Learning

Python uses a Global Interpreter Lock to make sure that  memory shared between threads isn’t corrupted. This is a design choice of the language that has it’s pros and cons. One of these cons is that in multi-threaded applications where at least one  thread applies a large load to the CPU, all other threads will slow down as well.

For multi-threaded Python applications that are at least somewhat time-sensitive, you should use Processes over Threads.

Experiment

I wrote a simple python script to show this phenomenon. Let’s take a look.

The core is this increment function. It takes in a Value and then sets it over and over, increment each loop, until the running_flag is set to false. The value of count_value is what is graphed later on, and is the measure of how fast things are going.

The other important bit is the load function:

Like incrementload is the target of a thread or process. The z variable quickly becomes large and computing the loop becomes difficult quickly.

The rest of the code is just a way to have different combinations of increment and load running at the same time for varying amounts of time.

Result

The graph really tells the story. Without the load thread running, the process and thread versions of increment run at essentially the same rate. When the load thread is running, increment  in a thread grinds to a halt compared to the process which is unaffected.

That’s all! I’ve pasted the full source below so you can try the experiment yourself.

Comparing blank string definition in Python3

In python3 using

or

Produces the same result for the programmer. Which one is faster? Using the python module timeit, it’s really easy to find out!

Using string="" is WAY faster.

Here’s the source code for my tests:

Creature Capture | Project Declaration & Top Level Flowchart

I’ve decided to embark on a video surveillance project! My family lives in a very rural part of the US, and constantly hear and see evidence of animals going crazy outside of my home at night. The goal of this project is to hopefully provide some kind of insight as to what animals actually live in my backyard.

Ideally, I want to monitor the yard using some kind if infrared motion detector. Upon a motion detection, an IR camera assisted by some IR spotlights would begin filming until it has been determined that there isn’t any more movement going on in yard. These clips would then be filed into a directory, and at the end of the night, they would be compiled and uploaded to YouTube. This video would then be sent to the user via email.

I’ve created the following flowchart to develop against as I begin implementing this idea.

I’ll be using a Raspberry Pi to implement this idea, a few months back I bought the IR camera module and haven’t used it for anything, this would be a good project to test it out.

There are a few hurtles that I’ll have to cross in order to make this project a success, like most groups of problems I deal with, they can be separated into hardware and software components.

Hardware

  1. Minimize false positives by strategically arranging motion detectors
  2. Make sure IR Spotlights are powerful enough to illuminate area
  3. Enclosure must be weatherproof & blend in with environment, Maine winters are brutal.

Software

  1. The Pi doesn’t have any built in software to take undetermined lengths of video.
  2. Must have a lot of error catching and other good OO concepts in order to ensure a long runtime.

I’ve actually come up with a routine for solving the first software problem I’ve listed, hopefully I’ll have an example of my solution in action later tonight.

Ideally, this project will have a working implementation completed by May 21, which is 7 days from now.

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.

PiPlanter 2 | Python Modules & Text Overlays

So in my last posting of the PiPlanter source code, the python script alone was 500 lines long. The intent with was to make things more modular and generic compared to the original version of the code that ran two years ago. Since the project has expanded a considerable amount since two summers ago, my goal of keeping everything short and concise isn’t really valid anymore so I’ve decided to split the code up into modules.

This improves a number of things, but it makes it kind of inconvenient to simply paste the full version of the source into a blog post. To remedy this, I’ll be utilizing www.esologic.com/source, something I made years ago to host things like fritzing schematics.

The newest publicly available source version can be found here: http://192.168.1.37/source/PiPlanter_2/ along with some documentation and schematics for each version to make sure everything can get set up properly. What do you think of this change? Will you miss the code updates in the body text of a blog post?

With all that out of the way, let’s talk about the actual changes I’ve made since the last post.

The first and foremost is that using Pillow, I’ve added a way to overlay text onto the timelapse frames like so:

Before

After

 

This was prompted by some strange behavior by the plants I noticed recently seen here:

I thought it was strange how the chive seemed to wilt and then stand back up and then wilt again, it would have been nice to be able to see the conditions in the room to try and determine what caused this. Hopefully I can catch some more behavior like this in the future.

Here is the new Image function with the text overly part included if you’re curious:

Now that I’ve got the PIL as part of this project, I’ll most likely start doing other manipulations / evaluations to the images in the future.

Okay! Thanks for reading.

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.

PiPlanter 2 | Adding Youtube Upload Functionality

In order to keep things moving quickly, I’ve decided to take a shortcut when it comes to uploading timelapse videos to youtube. I’ve decided to basically create a function that passes data to youtube-upload, a command line utility for linux that can upload videos very simply.

Here’s the function:

It should remind you a lot of “TryTweet” from the main version of the PiPlanter.

 

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.

PiPlanter 2 | Solving Broken Pipe Errors [Errno 32] in Tweepy

If I haven’t mentioned it already, https://twitter.com/piplanter_bot IS the new twitter account for PiPlanter. Like last time, I’m using the tweepy library for python to handle all things twitter for the project. What I’m NOT using this time is Flickr. From a design point of view, it wasn’t worth it. It was too complicated and had too many things that could go wrong for me to continue using it. Twitter is more than capable of hosting images, and tweepy has a very simple method of passing these images to twitter. Recently I moved the whole setup indoors and mounted it all onto a shelf seen here and it came with a set of strange problems.

Long story short, what I think happened was that since I moved them to a different location, the complexity of the images increased, causing an increase in the size of the images themselves. A broken pipe error implies that the entirety of the package sent to twitter wasn’t sent, causing the tweet not to go through. I first started to suspect this problem after seeing this:

 

The graphs were going through just fine, but images were seeming to have a hard time. You can’t tell from this photo, but those tweets are hours apart as opposed to the 20 minutes they are supposed to be. Once I started having this problem, I bit the bullet and integrated logging into my project which produced this log:

Hours and hours of failed tweets due to “[Errno 32] Broken pipe”. I tried a lot of things, I figured out that it was the size of the images after seeing this:

Photos that were simple in nature had no problem being sent. After scaling the image size down, I’ve had absolutely no problem sending tweets.


If you are tweeting images with tweepy in python and getting intermediate Broken pipe errors, decrease the size of your image.
Thanks for reading.

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.

PiPlanter 2 | Progress Update

I’m almost done with a very stable version of the Python code running the PiPlanter. There are many specific differences between this version of the python code and the version I wrote and implemented last summer, but the main one is that I tried to write functions for pretty much every task I wanted to do, and made each routine much more modular instead of one long line after line block to do each day. This took significantly longer to do (thus the lack of updates, sorry) but is much more expandable going forward. Below is the new version of the code, but by no means am I an expert programmer. The following code seems to work very well for what I want it to do.

Note the distinct lack of comments. I will put out a much more polished version of the code when it’s done. Before I move onto things like a web UI etc, I would like to do a few more things with this standalone version. The above version renders videos into time lapses, I would like to be able to upload those videos somewhere, hopefully youtube. I would also like to be able to email the log file to the user daily, which should be easier than uploading videos to youtube.

The script that renders the MySQL data into a graph is the following, it on the other hand has not changed much at all since last year and is still the best method to render graphs like I want to:

Here are some photos of the current setup, it hasn’t changed much since last time:

Thank you very much for reading.

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.

Server Upgrade | Stress Testing

So I wrote a program that makes really big numbers in python in an attempt to break a VM. Here’s the code:

I left it on for a few days and ended up with this:

Someone try and beat 26 compounds!

Hey! This post was written a long time ago, but I'm leaving it up on the off-chance it may help someone, but proceed with caution. It may not be a good idea to blindly integrate this code or work into your project, but instead use it as a starting point.