programmatic photography and automated video production solution
Imagine if you sleep peacefully while technology was hard at work programmatically photography vivid stars as the earth spins and beautiful Northern Lights dancing cross the sky. When the aurora were highly active, you would be wakened to step outside and enjoy every minute. Geeks in the Woods imagined this. They created the only solution. The timelapse videos below are some of the many that have automatically generated thus far.
The mission is to make photography programmatic and automatically generate the resulting photos into timelapse videos. While the camera takes photographs all day and streams them to the webcam page, the selected camera and lens are specifically for capturing vivid stars and active Northern Lights. The cameras used are the Sony Alpha mirrorless cameras with Sigma low light lenses.
no solutions for programmatic astrophotography
While the aurora can be forecasted to be visible with a high KPI, it may not be active where viewers are physically located. And the aurora usually becomes active early in the morning - usually from 2am to 6am in the morning. The Brown brothers do tend to work late but waiting up every night until 2am and beyond is not healthy. Thus, they wanted a system that could detect if the Northern Lights were actually visible from the property in Valdez, Alaska, while they and their guests were sleeping. They also want to share this unique Nordic experience with their friends and family around the world.
There are traditional webcams designed for outdoor applications like Nest Outdoor camera and aUniFi Video cameras. These traditional outdoor webcams even support low light, but do so by increasing the ISO and other night vision tactics. These traditional webcams do not have capability for adjusting the shutter speed to capture the stars vividly on a clear night or the aurora dancing cross the night sky. While the shutter speed is increased, the ISO should be as low as possible for ultimate clarity.
There are also video cameras that could be implemented to record the sky all night long. Video cameras are not designed for long exposures frames since they generally capture a specific number of frames per second. They would work when the aurora is extremely bright and highly active, they would not be able to capture stars via longer exposures. And capturing multiple frames per second over the course of hours and days creates lots of wasteful data when the objects are not moving at a rate that requires multiple frames per second.
The Brown brothers were intrigued by the ability to programmatically use Sony mirrorless cameras coupled with low light lenses to capture long exposure photographs, and thereafter, automatically generate timelapse videos from the long exposure photographs.
programmatic and automated solution
The solution to programmatically capture photographs using Sony Alpha cameras and turn these photographs automatically into timelapse videos is a distributed system relying on IOT devices and serverless technologies. The photographs are streamed real-time to the web and the timelapse videos are made available on the web immediately after their automated production.
In order to detect the aurora in photographs, an image recognition service provided by Amazon Web Services is used. If the photographs are tagged with "aurora", then the system will notify users opt-in to receive an alert of the aurora's immediate activity.
first automated photography
As lightness decreased and darkness returned, the Brown brothers were excited to continue experimenting astrophotography. The wanted to build a system to capture the stars and dreamed of capturing the aurora while they were sleeping. After discovering their Sony Alpha cameras had the ability to connect to them programmatically, Lee created the first script and begin testing out automated photography.
construct first weatherproof camera box
With the vast range of weather and its ability to change quickly from sun, to clouds, to snow, the Brown brothers needed to protect their Sony Alpha camera from the weather. They devised a plan to build an insulated box with a window on the front. The front window will protect the camera and keep the low light lens dry from precipitation.
The insulation along with a heater is intended to keep the electronics warm through the winter and prevent the cold from impacting the camera's performance. The heat is also intended to dry the front window - preventing rain and snow from collecting, causing the precipitation to appear in the resulting photograph.
success with automated photography
The Python automated photography scripts originally ran on an Apple MacBook with OSX. Since OSX comes standard with the ability to run Python, this was frictionless way to build, test and iterate upon the prototype.
The script was finally harden over many days and nights resolving issues and making enhancements, and now was reliably capture photographs automatically and streaming them to the Webcam page on geeksinthewoods.com.
deploy photography script to Raspberry Pi
October 6th, 2018
Since the automated photography script was successfully running on the MacBook, it was time to migrate it to a simplified device that would be less expensive to risk operating outside in Alaska's sometimes extreme weather even with our best abilities to make a weatherproof box. Surprisingly, deploying and operating the script on the Raspberry Pi was surprisingly easy.
Construction of 3D printed Weatherproof Camera Box begins
October 8th, 2018
Bradley Pizzimenti designs CAD model for weatherproof camera box. On this day, he begins 3D printing printing the first model.
The programmatic photography system is comprised of a Sony Alpha camera and a Raspberry Pi. Since the Raspberry Pi runs Debian linux, it comes standard with Python 2. Thus, the script responsible for the programmatic photograph is written in Python and deployed to the Raspberry Pi device.
Additional deployments of a Sony Alpha camera and Raspberry Pi with the programmatic photography script can be deployed for specific views. The diagram below shows how there are two deployments: one positioned to capture the Northern Lights and the other positioned to capture daytime over the lake.
The Python script is in charge of connecting to the camera, taking a photography with optimal shutter speed, and then transferring the photograph from the camera to the Raspberry Pi. The script also provides a management interface available by connecting to the Raspberry Pi with a web browser to allow viewing the latest image and stop/start the capture.
The Python programmatic photography script takes photographs at specified interval. The photography interval is every 60 seconds. By default, the script tries to first take a photography at 200 ISO. If the photograph is too dark, the script increased the shutter speed to reach a clear photography at 200 ISO. If the shutter speed has to be increased greater than 20 seconds while ISO is at 200, then ISO is increased to an appropriate value. Since Sony Alpha mirrorless cameras have max shutter speeds of 20 seconds, if a shutter speed length above 20 seconds is required, then the script set camera to use Auto ISO. (Note that the max shutter speed for the Sony A6300 camera is actually 30 seconds, but any longer than 20 seconds will cause star trailing.)
The Python script uses Sony's Camera Remote API beta. The sad news is that Sony will not longer be providing updates and active support for Camera Remote API. Sony Alpha Remote Camera API. Thus, this programmatic photography script will only work with Sony A7, Sony A7 II, Sony A6300 and Sony A6000 mirrorless cameras.
The Raspberry PI is connected to a local Wifi network. The script deployed on the Raspberry Pi uses the device's Internet connection to upload each photograph after is capture to AWS S3 for cloud storage and web accessibility. These programmatic photographs are made available in real-time GeeksInTheWoods.com on the Webcam page.
photo processing and aurora detection
The Photo Processing system is responsible for taking the photographs, including those of the aurora and stars, conducting appropriate image enhancements and using AWS's image recognition service to tag the photographs with appropriate keywords. The functionality includes detecting when the aurora in a photography and notifying users by distristibuting alerts.
The functionality to detect the aurora uses AWS's image recognition service. While processing the photograph, the photograph is sent to the AWS image recognition service. The service returns keywords that were identified as part of the photography. The keywords are associated with the photography by include them as tags as part of the meta data for the file.
If a photograph is tagged with "aurora" and the level of confidence is above 90%, then the system knows the aurora is visible. This visibility of the aurora triggers alerts to be generated. The alerts are sent to users that have opted-in via a text message and even a voice phone call to notify them to wake up and get outside to see the aurora.
Once the photographs have been programmatically enhanced including color correction, they are uploaded to AWS S3 where they are stored and made accessible on the web. These photographs can be seen at the bottom of the Webcam page and these photographs are then used for automated video production.
automated video production
The Automated Video Production system is responsible for taking the processed photographs and generating timelapse videos. The process runs three times per day creating four videos:
- Once after sunrise for the day to create two timelapse videos. One production is the sunrise timelapse videos, taking photographs from one hour prior to sunrise to one hour after sunrise. The second production is the night timelapse videos, taking photographs from after sunset of the prior day to sunrise of the current day.
- Once after sunset, taking photographs from one hour prior to sunset to one hour after sunset. These photographs create the sunset timelapse videos.
- Once after midnight, taking all photographs from prior day to create timelapse video for the day.
The autogenerated timelapse videos are uploaded to AWS S3 for cloud storage and web accessibility. While all the autogenerated timelapse videos are posted on this website, the best videos are published to our Geeks in the Woods YouTube channel.
Since the Brown brothers are entrepreneurial software engineers, they prefer focusing on creating solutions with code and not wasting precious time managing servers. Thus, they are excited with the evolution and growing trends of serverless architecture.
As you can see in the above diagram, Photo Processing, Autogenerated Video Production, and along with the API and web app are deployed on AWS Fargate rather than on AWS's old standard of EC2 instances. While EC2 instances are provide access to the VM, Fargate only allows access to the task. Thus, deploying to EC2 only abstracts to the hardware and requires software engineers or their IT ops teams to deploy and manage servers before software can be deployed and managed. Fargate abstracts the hardware level and abstracts to the task level. Software engineers no longer need to provision and manage servers before deploying and releasing software. With Fargate, software engineers can focus on deploying and releasing the tasks their software was designed to achieve.
Lee Brown, co-founder of Geeks in the Woods, shares his experience with Serverless Architecture in October 2018 in the presentation below.
weatherproof camera box
Electronics usually are not built for extreme weather conditions and cold temperatures do not generally make them reliable. In addition, the camera and the lens need to be protected from precipitation. Since there are no commercially available weatherproof boxes for cameras, the Brown brothers has to build their own. The basic needs for the box is outlined in this diagram:
The Brown brothers created their first weatherproof camera box out of wood and insulation board. After several enhancement and iterations, the completed prototype was quite large, but enabled them to determine more specific requirements for a better weatherproof camera box.
Accordingly, an outdoor box for the camera for Alaska's extreme weather conditions needs to be designed to meet these requirements:
- front glass window to protect camera and lens from both rain and snow
- sealed to prevent rain and melting snow from leaking into the interior
- dimensions to provide space for 1 Sony Alpha mirrorless camera
- mount with inset 0.25" screw fixed to floor to secure camera
- ability to open to install and conduct maintenance on the equipment
- space for Raspberry Pi device
- ability for heat pad to be installed on floor and it will not melt
With an understand of the perfect weatherproof camera box, they worked with Bradley Pizzimenti to turn the concept into a reality. This perfect weatherproof camera box will be 3D printed in Anchorage.