Search This Blog

Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

Tuesday, 9 October 2018

Creating a very basic python flask app in a docker container

Instructions for creating a very basic "hello flasky" Python Flask application in a docker container.  These instructions are for hand building the app on a single host and have no niceties (at the moment) for things like WSGI, NGINX, CI/CD, etc.  I put them together when I was learning the very basics of how to create a container.

Step 1 - create a host.

Create a host VM to run docker on.  As it happens, I have an old Windows8 PC that I run as a HyperV server, but any server would do.  I assigned my VM a single core and 1.5Gb of RAM which has been more than adequate for this project.

  • Install Ubuntu18.04 server - select "Docker" option during install
  • Set a static IP (or use DNS if you have some locally)


Step 2 - create a container.

I created a Flask app that simply returns "HELLO FLASKY" when queried by a web browser.  I adapted the instructions from here

SSH into your VM (I used putty on a Windows PC, but use whatever floats your boat)

Run the following bash commands from console:
 mkdir ~/helloflasky  
 cd ~/helloflasky  
 sudo nano helloFlasky.py  

Enter python code into nano:
 from flask import Flask  
 app = Flask(__name__)  
 @app.route("/")  
 def greeting():  
   return "<h1 style='color:green'>Hello Flasky!</h1>"  
 if __name__ == "__main__":  
   app.run(host='0.0.0.0')  
...save and exit nano

If python tools are installed locally (python3, python3-dev, python3-pip, flask), test the app on http://localhost:5000

Create a Dockerfile using nano:
 sudo nano Dockerfile  

Enter dockerfile into nano:
 FROM python  
 LABEL maintainer="Nik"  
 RUN apt-get update  
 RUN apt-get install -y python3 python3-dev python3-pip nginx  
 RUN pip3 install uwsgi  
 RUN pip3 install flask  
 COPY ./ ./app  
 WORKDIR ./app  
 CMD python3 helloFlasky.py  
...save and exit nano

Build container image
 sudo docker build -t hello-flasky .  

Create a container bound to TCP:5000
 sudo docker run -p 5000:5000 -d hello-flasky  

Check it using a web browser on http://host_ip:5000


NEXT STEPS :

  • Run the app using uwsgi or gunicorn.  
  • Wrapper with NGINX.
  • Create a data container (probably Mongo)
  • Have helloFlasky get data from second container
  • CI/CD




Further container thoughts

MONGO

Create a folder ~/data

Created a Mongo container
docker run -d -p 27017:27017 -v /home/nik/data:/data/db mongo

Tested with Robo 3T client (Windows)

Friday, 13 January 2017

Wrapping a python script for "live" use

I have a internal python app that I want to expose to people on my LAN.  It doesn't need to be enterprise level HA or anything, but I'd like it to have good (i.e. >98%) availability and behave like proper app most of the time and do stuff like starting properly follow reboots.

Create a service to run the script


On CentOS 7, use systemd to create the service

Adapted from article here:
http://www.raspberrypi-spy.co.uk/2015/10/how-to-autorun-a-python-script-on-boot-using-systemd/

(updated 24/09/2018 with restart setting based on this article https://singlebrook.com/2017/10/23/auto-restart-crashed-service-systemd/ )

sudo nano /lib/systemd/system/myscript.service


Description=Garage
ServiceAfter=multi-user.target

[Service]
Type=idle
WorkingDirectory=/home/pi/garage
ExecStart=/usr/bin/python3 /home/pi/garage/runserver.py > /home/pi/garage/garage.systemd.log 2>&1
StandardOutput=syslog
StandardError=syslog
Restart=always

RestartSec=10


[Install]
WantedBy=multi-user.target




sudo systemctl daemon-reload
sudo systemctl enable myscript.service
sudo reboot


nginx

Install nginx with yum

Config file (/etc/nginx/nginx.conf)


# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        #listen       [::]:80 default_server;
        server_name  www.whateveryoulike.com;

        # Load configuration files for the default server block.
        #include /etc/nginx/default.d/*.conf;

        location / {
                proxy_pass http://localhost:5555;
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}


If there is a 502 Bad Gateway error, it may be secure Linux settings in CentOS :

sudo /usr/sbin/setsebool httpd_can_network_connect true -P


Firewalld


sudo firewall-cmd --get-active-zones
public
  interfaces: eth0

sudo firewall-cmd --zone=public --add-service=http --permanent

sudo firewall-cmd --reload


Monday, 12 December 2016

Some Docker stuff

I want to extend my Xmas Lights project so I can record when the lights have been on/off.  I figure a database is the best way to do this, so not wanting to learn something like Mongo at this point, I figured a Postgres DB was probably easiest.  Not wishing to get into the installation process I wondered whether Docker might hold a quick solution to my problem.  Here are a few things I learned.

Platform:
Windows 8.1 running HyperV - LAN IP 192.168.0.60
Linux VM : "CentOS Linux release 7.2.1511 (Core)", logged in as "root"

Setup docker
sudo yum install docker

Configure host firewall
firewall-cmd --permanent --zone=trusted --change-interface=docker0
firewall-cmd --permanent --zone=trusted --add-port=4243/tcp
firewall-cmd --reload
(interestingly, I found that I couldn't start containers with firewalld disabled)

Create Postgres container
docker run --name postgres -p 5432:5432 -d postgres
(I should really use "-e POSTGRES_PASSWORD=xxx", but omitting it seems to just allow the "postgres" login to work with a blank password)
docker inspect postgres
(output contains IP address : "IPAddress": "172.17.0.2".  This is the Docker internal address.  It appears that by default other containers on this internal network can see each other)

Create PGAdmin4 container
docker run --name pgadmin --link 1337c2af44ac:postgres -p 5050:5050 -d fenglc/pgadmin4
(not sure about the --link bit, I think the version of Docker I'm using is much newer and handles internal networking differently, but this command worked OK)

Use WebUI to create a database "xmas16" and a table "messages" with columns ID(integer) and message(text).  Put some test data in it.

Web browser to http://192.168.0.60/browser/ (IP of host CentOS VM) to access PGAdmin console.  Connected PGAdmin to Postgres server "172.17.0.2"

In Python, installed psycopg2 module using PIP

Python code (running on separate test PC or RPi):

import psycopg2
conn = psycopg2.connect("dbname='xmas16' user='postgres' host='192.168.0.60' password=''")
cur = conn.cursor()
cur.execute("select * from messages")
rows = cur.fetchall()
for row in rows:
    print (row[1])

Done!  My app wrote my test messages to the stdout.  Learning Docker took an hour or two, especially the firewalld bit, but I have a full PoC rig up and running with very little effort.  Considering my objective is to capture non-sensitive data from a Xmas light project for fun it's more than adequate.

Tuesday, 15 November 2016

Xmas lights controller 2016

Last year I decided to use a Raspberry Pi and a relay board to switch my Christmas lights on and off.  This had two benefits; the first is that the kids thought it was super cool, especially my niece who turned on our lights from the other end of the country while watching it on Skype, and secondly it became straightforward to schedule them to switch on and off at multiple different times each day.

Anyway, it's now mid November and it's gotten too cold and dark for it to be fun outside after work so I've been looking for indoors projects to do, which inevitably led to the Pi.  During the last year I've been playing with a bit more Python (a welcome break from the whole lotta Powershell we've been doing), and have recently touched on Cordova for mobile apps, so wanted to look at something which might tie in with that, so I decided to resurrect the Xmas Lights project for this year, but build it with a bit more care this time around.


I've been hacking away for the last couple of evenings, and have got the board into what I think is reasonable shape.  It's a bit early in the year to have the Christmas lights on outside still, so this post just covers the Pi and relay board.  I might make a YouTube video nearer Christmas when i plug it all in.


So, here's the board:






It's an "8 Channel 5V Active Low Relay Module Board" which sounds advanced but is actually the cheapest piece of crap I could find on eBay at the time and cost £4.28.  That said, it's been fine, although it's not earthed or particularly well assembled, so I'm not sure I'd want to run 240v through it...  The black box is a Pi A+ that I've had kicking around for a while since I bricked my original B and haven't soldered the headers onto my Zero

You might be able to see in the pictures, but the pin mapping is:

RelayBoard   Pi-pin  GPIO
GND <--------> 6  <--> Ground
IN1 <--------> 37 <--> 26
IN2 <--------> 35 <--> 19
IN3 <--------> 33 <--> 13
IN4 <--------> 31 <-->  6
IN5 <--------> 29 <-->  5
IN6 <--------> 40 <--> 21
IN7 <--------> 38 <--> 20
IN8 <--------> 36 <--> 16
VCC <--------> 2  <--> +5v

Like last year, the project is based on Python Flask.  The folder structure and code are at the end of this post.

The main differences since last year in views.py are that it uses more HTTP verbs in the bindings, uses dictionary objects to manage the pins states and returns proper JSON documents to communicate state requests between the client and server.  The other big difference is in lights.html that is now using bootstrap (so my site can look just like everyone else's...), much smarter AJAX to send state requests and a UI to render the current state based the server response from each request.  

As the state is actually derived from querying the GPIO pins for their value, this is clearly not thread safe in any way so I'm not sure there's a huge amount of value in taking the UI further, but I might look into using web sockets or something and pushing updates to clients in future.

I'll have to look into the finer points of the weird pseudo JS that cordova uses, but if it's straightforward to translate my Javascript then I might also dip my toe in the waters of "native" mobile apps.




run.py
|-app
  |-templates
  | |-help.html
  | |-lights.html
  |-__init__.py
  |-views.py

and here is the code:

run.py
#!/usr/bin/env python
#!flask/bin/python
from app import app
app.run(debug=True, host='0.0.0.0', port=80)
__init__.py
from flask import Flask
import RPi.GPIO as GPIO
app = Flask(__name__)
from app import views
GPIO.setwarnings(False)
GPIO.setmode(GPIO.BCM)
pins = {"26":1, "19":1, "13":1, "6":1, "5":1, "21":1, "20":1, "16":1}
for p in pins:
GPIO.setup(int(p), GPIO.OUT)
GPIO.output(int(p), pins[p])
views.py
from app import app
from flask import request
import RPi.GPIO as GPIO
from flask import render_template
import time
import json
mapping = {"a":26, "b":19, "c":13, "d":6, "e":5, "f":21, "g":20, "h":16}
def getState():
relays = {"a":"x", "b":"x", "c":"x", "d":"x", "e":"x", "f":"x", "g":"x", "h":"x"}
for m in mapping:
if GPIO.input(mapping[m]) == 1:
relays[m] = "off"
else:
relays[m] = "on"
return relays
def setState(newState):
for s in newState:
if s in mapping:
if newState[s] == "on":
GPIO.output(int(mapping[s]), 0)
time.sleep(0.2)
elif newState[s] == "off":
GPIO.output(int(mapping[s]), 1)
time.sleep(0.2)
elif newState[s] == "toggle":
if getState()[s] == 'on':
setState({s:"off"})
else:
setState({s:"on"})
time.sleep(0.2)
else:
#newState contains unknown relay
print "err"
#def setPins(pinVal):
# pins = [26, 19, 13, 6, 5]
# for p in pins:
# GPIO.output(int(p), pinVal)
# time.sleep(0.1)
@app.route('/lights', methods=['PUT'])
def lights_on():
setState({"a":"on", "b":"on", "c":"on", "d":"on", "e":"on", "f":"on", "g":"on", "h":"on"})
return json.dumps(getState())
@app.route('/lights', methods=['DELETE'])
def lights_off():
setState({"a":"off", "b":"off", "c":"off", "d":"off", "e":"off", "f":"off", "g":"off", "h":"off"})
return json.dumps(getState())
@app.route('/lights', methods=['POST'])
def lights_set():
setState(request.form)
return json.dumps(getState())
@app.route('/', methods=['GET'])
def jsLights():
return render_template('lights.html')
@app.route('/status')
def status():
return json.dumps(getState())
@app.route('/help')
def help():
return render_template('help.html', relays=getState())

lights.html
<html>
<head>
<title>Xmas Lights 2016</title>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" />
<style>
.row-buffer{ margin-bottom: 20px; }
</style>
<script>
function doAlert(lightsStatus){
var obj = jQuery.parseJSON(lightsStatus);
$("#lightsA").removeClass("btn-warning");
$("#lightsB").removeClass("btn-warning");
$("#lightsC").removeClass("btn-warning");
$("#lightsD").removeClass("btn-warning");
$("#lightsE").removeClass("btn-warning");
$("#lightsF").removeClass("btn-warning");
$("#lightsG").removeClass("btn-warning");
$("#lightsH").removeClass("btn-warning");
if(obj.a=="on"){ $("#lightsA").addClass("btn-warning") }
if(obj.b=="on"){ $("#lightsB").addClass("btn-warning") }
if(obj.c=="on"){ $("#lightsC").addClass("btn-warning") }
if(obj.d=="on"){ $("#lightsD").addClass("btn-warning") }
if(obj.e=="on"){ $("#lightsE").addClass("btn-warning") }
if(obj.f=="on"){ $("#lightsF").addClass("btn-warning") }
if(obj.g=="on"){ $("#lightsG").addClass("btn-warning") }
if(obj.h=="on"){ $("#lightsH").addClass("btn-warning") }
}
$(document).ready(function() {
$("#lightsOn").click(function() {
$.ajax({url: "http://192.168.0.50/lights", type: "PUT"}).done(function(data) {
doAlert(data);
});
});
$("#lightsOff").click(function() {
$.ajax({url: "http://192.168.0.50/lights", type: "DELETE"}).done(function(data) {
doAlert(data);
});
});
$("#lightsA").click(function() { $.ajax({url: "http://192.168.0.50/lights", type: "POST", data: {"a":"toggle"} }).done(function(data) {doAlert(data)}) });
$("#lightsB").click(function() { $.ajax({url: "http://192.168.0.50/lights", type: "POST", data: {"b":"toggle"} }).done(function(data) {doAlert(data)}) });
$("#lightsC").click(function() { $.ajax({url: "http://192.168.0.50/lights", type: "POST", data: {"c":"toggle"} }).done(function(data) {doAlert(data)}) });
$("#lightsD").click(function() { $.ajax({url: "http://192.168.0.50/lights", type: "POST", data: {"d":"toggle"} }).done(function(data) {doAlert(data)}) });
$("#lightsE").click(function() { $.ajax({url: "http://192.168.0.50/lights", type: "POST", data: {"e":"toggle"} }).done(function(data) {doAlert(data)}) });
$("#lightsF").click(function() { $.ajax({url: "http://192.168.0.50/lights", type: "POST", data: {"f":"toggle"} }).done(function(data) {doAlert(data)}) });
$("#lightsG").click(function() { $.ajax({url: "http://192.168.0.50/lights", type: "POST", data: {"g":"toggle"} }).done(function(data) {doAlert(data)}) });
$("#lightsH").click(function() { $.ajax({url: "http://192.168.0.50/lights", type: "POST", data: {"h":"toggle"} }).done(function(data) {doAlert(data)}) });
});
</script>
</head>
<body>
<h1>Xmas 2016 lights control</h1>
<div class="row row-buffer">
<div class="col-sm-8">
<button id="lightsOn" type="button" class="btn btn-primary btn-block">All lights on</button>
</div>
</div>
<div class="row row-buffer">
<div class="col-sm-8">
<button id="lightsOff" type="button" class="btn btn-primary btn-block">All lights off</button>
</div>
</div>
<div class="row">
<div class="col-sm-1">A</div>
<div class="col-sm-1">B</div>
<div class="col-sm-1">C</div>
<div class="col-sm-1">D</div>
<div class="col-sm-1">E</div>
<div class="col-sm-1">F</div>
<div class="col-sm-1">G</div>
<div class="col-sm-1">H</div>
</div>
<div class="row">
<div class="col-sm-1"><button id="lightsA" type="button" class="btn btn-lg btn-block"><span class="glyphicon glyphicon-off"></span></button></div>
<div class="col-sm-1"><button id="lightsB" type="button" class="btn btn-lg btn-block"><span class="glyphicon glyphicon-off"></span></button></div>
<div class="col-sm-1"><button id="lightsC" type="button" class="btn btn-lg btn-block"><span class="glyphicon glyphicon-off"></span></button></div>
<div class="col-sm-1"><button id="lightsD" type="button" class="btn btn-lg btn-block"><span class="glyphicon glyphicon-off"></span></button></div>
<div class="col-sm-1"><button id="lightsE" type="button" class="btn btn-lg btn-block"><span class="glyphicon glyphicon-off"></span></button></div>
<div class="col-sm-1"><button id="lightsF" type="button" class="btn btn-lg btn-block"><span class="glyphicon glyphicon-off"></span></button></div>
<div class="col-sm-1"><button id="lightsG" type="button" class="btn btn-lg btn-block"><span class="glyphicon glyphicon-off"></span></button></div>
<div class="col-sm-1"><button id="lightsH" type="button" class="btn btn-lg btn-block"><span class="glyphicon glyphicon-off"></span></button></div>
</div>
</body>
</html>
help.html
<html>
<head>
<title>Xmas Lights 2016 - Help</title>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" />
</head>
<body>
<h1>
Xmas2016 lights controller - Help
</h1>
<div>
Usage:
<table class="table table-bordered">
<tr>
<td>switch all on</td><td>PUT root/lights</td>
</tr>
<tr>
<td>switch all off</td><td>DELETE root/lights</td>
</tr>
<tr>
<td>control relays</td><td>POST {{relays}} root/lights</td>
</tr>
</table>
</div>
</body>
</html>


Finally a picture I took while I was messing around with my camera - you can clearly see the opto-isolators are not even vaguely straight - not a quality of board to be handling voltages and currents that could burn my house down I think.