Category Archives: Uncategorized

WTFisThisRegister, Beginner’s Flask App: Dictionary based on Flaskr (uses sqlite3) and deployed to Heroku

Over the course of a night and morning I built a small flask app (Flask is a micro web framework for Python. Django is similar but more heavy-duty). It is essentially a dictionary. You can search for an entry by its keyword, or view the helptexts for all the entries, or (if you are logged in) add and remove entries. It is a straightforward derivative of excellent the Flaskr tutorial.

For me, I am really proud of this app even though it is dirt simple and not much to look at because this is the first time I really interacted with a database in a programming language.

The intent was to make a database of documentation for the AVR Atmega328P microcontroller. However, since then I have decided to move to using a mediawiki (sample page).

The app may be seen live at salty-retreat-5363.herokuapp.com. The username is “admin” and the password is “default”. The database gets wiped at least once every 24 hours.

If you’d like to run the app locally and play around with source code, please see https://github.com/nouyang/WTFisThisRegister for detailed instructions. That page also details how to deploy this app (or similar ones, such as Flaskr) to Heroku. I used http://dillinger.io/ to play around with markdown online and learn it and actually document this project with a proper README.

Things Learned

Here are some things I learned in the course of making this:

  1. Gitignore files need to be created before you run “git add .” — in general, the line telling it to ignore a file needs to exist before you added it to the version control, git won’t automagically remove it for you.
  2. Heroku DOES play with sqlite3 but not happily, for instance the database will be wiped at least once every 24 hours. See https://devcenter.heroku.com/articles/sqlite3.
  3. Virtualenv does NOT like spaces in your directory path! This was tricky to debug as even in -v verbose mode the full path for my virtualenv packages was shortened and so I couldn’t even notice that my path had spaces in it.
  4. Databases use cursors.

The Process

Here are some images from the process. Click on them to view them in full resolution.

1. First I started out just using a dictionary (no database involved) and a single hello.py file.

2. I then followed the Flaskr tutorial. After doing so I wanted to modify it so that the text was monospaced and preserved whitespaces, because it doesn’t by default.

I used the “pre” tag at first.

but soon switched to using CSS.

This actually caused me a lot of headache later troubleshooting why there was random whitespace in my design, because I didn’t restrict where the CSS was applied enough. So much whitespace:


 In the end I had this file: https://github.com/nouyang/WTFisThisRegister/blob/master/static/style.css

.helptext { preserve whitespace with wrapping CSS }

and the corresponding code in the html file was

 <div class="helptext"%gt;
{{ entry.helptext|safe }}</div>

3. I wanted to be able to delete entries, so I added a delete button.

In show_entries.htmloops can’t show it because blogger can’t escape this HTML properly :/ In the main python file:

@app.route('/delete', methods='POST'])                                                                                         
def delete_entry():
if not session.get('logged_in'):
abort(401)
g.db.execute('delete from entries where keyword = ?', [request.form['entry_to_delete']])
g.db.commit()
flash('Entry was successfully removed')
return redirect(url_for('show_entries'))

The “?” is a safe why for sqlite3 to accept text (in this case, whatever was entered in the form with the name entry_to_delete).

4. I learned to play around with the database to test my SQL commands in the terminal:

 $ sqlite3 /tmp/flaskr.db
sqlite> select helptext from entries where helptext='PCMSK0'

5. Finally I added a search functionality to the site:

@app.route('/search', methods=['POST'])                                        
def search_entries():
keyword = request.form['searchterm']
cur = g.db.execute('select helptext from entries where keyword = ?',
[keyword])
result = [dict(keyword=keyword, helptext=row[0]) for row in cur.fetchall()]
return render_template('search.html', result=result)

And set the homepage to be the search page:


@app.route('/')
def show_search():
return render_template('search.html')

6. I learned about initializing databases, namely by dumbly thinking I needed to include the flaskr.db file and no schema.sql file in the repository to allow other people to get it working on their systems. Actually, I was just missing a line, “init_db()”, before “app_run()”. In the end I am still keeping it without that line so that I have persistence in the database (I can stop the server and restart it and the database entries will still be there).

All in all I learned a lot! Do browse through the source code and play it with yourself if you’d like.

 =========================================

Liveblog Notes (kept here for reference)

http://flask.pocoo.org/docs/quickstart

http://maximebf.com/blog/2012/10/building-websites-in-python-with-flask/
https://github.com/trtg/flask_assets_tutorial

(did not use: http://www.realpython.com/blog/python/python-web-applications-with-flask-part-ii-app-creation/)

http://flask.pocoo.org/docs/tutorial/introduction/ Ah! Here’s the hands-on quickstart.

Hmm, well now the newlines show up but long lines aren’t wrapped.
To fix:
http://stackoverflow.com/questions/642140/retain-newlines-in-html-but-wrap-text-possible

.page { width: 70%;
min-width:35em;
border-radius:5px;
}

Modifying the database name: How are you supposed to properly do this? I just deleted /tmp/flaskr.db and then ran in the shell:
sqlite3 /tmp/flaskr.db < schema.sql

and things seemed to work again.
Include a delete button in the future:
If you don’t want your data wiped clean, DO NOT do 
$ python
>>> from flaskr import init_db
>>> init_db()
Instead, to play with sqlite3 commands,
sqlite3 /tmp/flaskr.db
And then use the same commands as in flask, but with semicolons.
So to lookup something we need the WHERE command:
e.g. for me it is:
select helptext from entries where keyword=”PCMSK0″;
Uhm, okay, now how to display it in my flask app?
ahh what are database i am so confused
http://www.zetcode.com/db/sqlitepythontutorial/
http://docs.python.org/2/library/sqlite3.html

Let’s install the AVR eclipse plugin
http://avr-eclipse.sourceforge.net/wiki/index.php/The_AVR_GCC_Toolchain#Debian_and_Ubuntu
http://www.instructables.com/id/How-to-get-started-with-Eclipse-and-AVR/step13/Quick-Tour-Of-Cool-Features/ (windows)

sudo apt-get install eclipse (270+ MB! sigh)

sudo apt-get install gcc-avr binutils-avr gdb-avr avr-libc avrdude

http://avr-eclipse.sourceforge.net/wiki/index.php/Plugin_Download

Heroku deploy
http://flask.pocoo.org/docs/quickstart/#quickstart-deployment
https://devcenter.heroku.com/articles/getting-started-with-python

  1. Install heroku toolchain
  2. Login to heroku
  3. Install and activate virtual environment
  4. Install dependencies (why do I need gunicorn? not sure) 
  5. Make a Procfile
  6. Run foreman and check in browser that app is up on localhost
  7. Download the python .gitignore and add the “venv” line to the top
  8. my own step: add to github, git remote add origin https://github.com/nouyang/WTFisThisRegister.git

https://help.github.com/articles/ignoring-files
OH. The .gitignore file does NOT go in the root directory directly. No wonder everything failed and I just had to delete my github repo and also the .git folder in my root directory.

Instead,

For example, you might create the file at ~/.gitignore_global and add some rules to it. To add this file to your cross-repository configuration, run git config --global core.excludesfile ~/.gitignore_global. Local per-repository rules can be added to the .git/info/exclude file in your repository. This method can be used for locally-generated files that you don’t expect other users to generate.

Wait, but what if there’s one file, e.g. a passwords for the database file, that I don’t want to commit? Maybe it’s just messed up because I did the git init and then added the gitignore file or something funky, I don’t remember. http://stackoverflow.com/questions/1139762/gitignore-file-not-ignoring

Okay, so now I want to pull the config parameters (passwords, admin usernames, secretkeys) out into a separate file that is gitignored.
Done. Created a file called databaseconfig.py, added the line “from databaseconfig import secretkey, username, password” to the main python file.

Oh I should generate a real secret key
http://stackoverflow.com/questions/18247971/print-python-os-urandom-output-on-terminal
python
>>> import os, binascii
>>> binascii.hexlify(os.urandom(24))

Whoa they actually mean create the file ~/.gitignore_global, not create ~/.gitignore_global/.gitignore
Whoops.

http://stackoverflow.com/questions/4824188/git-ignore-vim-temporary-files
Also, vim and gedit create swap files not covered by github’s gitignores, so add

*~

and

*.swp
*.swo

to ~/.gitignore_global

Yes! Succesfully used .gitignore for once.

(venv)nrw@nrw-PC:~/projects/WTFisThisRegister$ git commit -m “init”
[master (root-commit) 35b1400] init
 10 files changed, 228 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 Procfile
 create mode 100644 README.md
 create mode 100644 WTFisThisRegister.py
 create mode 100644 requirements.txt
 create mode 100644 static/style.css
 create mode 100644 templates/layout.html
 create mode 100644 templates/login.html
 create mode 100644 templates/search.html
 create mode 100644 templates/show_entries.html

Now add it to github.
git remote add origin https://github.com/nouyang/WTFisThisRegister.git
git push -u origin master
Okay, now to finish the heroku deploy.
heroku create 
git push heroku master

Permission denied (publickey).
fatal: The remote end hung up unexpectedly

Whoops I need to add this computer to heroku.
https://devcenter.heroku.com/articles/keys
$ ssh-keygen -t rsa
/home/nrw/.ssh/id_rsa already exists.

Whoops, let’s not overwrite that, maybe it will break github. 
cat ~/.ssh/id_rsa.pub
and copy that into heroku under Account (https://dashboard.heroku.com/account).
Alright, let’s try pushing to heroku again.

—–> Launching… done, v3
       http://salty-retreat-5363.herokuapp.com deployed to Heroku

Yay!!
Oh noes. It doesn’t work on heroku… Is that because I gitignored the databaseconfig.py file and so it doesn’t exist on heroku?
Guhuhuh I guess I should learn to use github branches.
Okay nevermind, I found a better answer than having two branches to maintain and update if I ever change anything (bletch).
This gives a clear explanation and example: https://devcenter.heroku.com/articles/config-vars
So in the main python file I put
SECRET_KEY = ENV[‘SECRETKEY’]
and then in the terminal I type
heroku config:set SECRETKEY=blahblah.
Then, git add, git commit, and then git push heroku master.
NOPE that didn’t fix it ;____; wahh I am sad.
Let’s research….

http://stackoverflow.com/questions/18037027/flask-app-failing-on-heroku-but-working-with-foreman
WHOA good point I forgot to turn debug off! Whaaaa. Okay that is terrible of me.
# configuration
DEBUG = False
WAH. still sad.
Let’s try to fix foreman by adding the var to .env… Does that work?
.env does not like spaces around the equal sign assignment operator!
S3_KEY=mykey
NOT 
S3_KEY = mykey
Oh! Make sure to add “import os” to the main python file if we are using os.environ. DUH.

Note: Only today I realized that in pasting URLs from the location bar in chromium-browser (ubuntu 12.10) into blogger, the URLs are automatically turned into hyperlinks, but they remain plain old text and I have to manually turn them into hyperlinks when I copy them from firefox, my main browser 🙁 Such a weird specific bug.

Ah? It still doesn’t work?? Why? It works locally on foreman, but not remote on heroku.

Okay, let’s look at the heroku logs.
https://devcenter.heroku.com/articles/logging
ImportError: No module named databaseconfig

Whoops. So a gitignore issue.
UGH. Everything becomes 10x more complicated when working in a virtual environment. Fine. I’ll take the hit and jump full out into non-localhost environment, where I can’t just run “python blah.py” I have to run “foreman start” and other issues. Because foreman looks for a “.env” file. But python does not. (http://stackoverflow.com/questions/12335488/cannot-use-environment-variables-for-settings-in-django)

The solution is to use os.environ.get() instead.

Okay, so I’ll just write a short if statement checking if os.environ.get() is None. 

SECRET_KEY = os.environ.get(‘SECRETKEY’) #for heroku deploys.                
if SECRET_KEY == None:                                                      
    SECRET_KEY = secretkey    

Great, now my config works in with both “foreman start” and “python WTFisThisRegister.py”.

Now to get databaseconfig to show up in heroku. Do I want to?
No. A cleaner solution instead of all this is to use try, importerrors. Since the present of databaseconfig.py can be used to indicate that we are running it locally.

ahhhh now the /entries works locally but there is an internal server error remotely. WHY.
http://stackoverflow.com/questions/8950674/debugging-a-flask-app-running-in-gunicorn

Instead of foreman start, just use foreman run python app.py if you want to debug your application in development.

 Maybe it’s an issue with gunicorn or procfile or whatever that stuff is?
 web: gunicorn WTFisThisRegister:app        
http://ryaneshea.com/lightweight-python-apps-with-flask-twitter-bootstrap-and-heroku
changed to
web: python WTFisThisRegister.py

Okay, still works locally but not remotely. Maybe it is a port issue. Sucks that these tiny changes debugging production (heroku environment) take forever to try (30 seconds for each time I git push heroku master)

2014-01-12T05:58:48.956357+00:00 heroku[web.1]: State changed from crashed to starting

Yay! Fixed ports issue.

AGH. Still internal server error. WTF.
http://stackoverflow.com/questions/7653010/import-sqlite3-with-python2-7-on-heroku
So sqlite3 could NOT be used on the previous version of heroku. However, I think it should be able to now?
https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem
http://stackoverflow.com/questions/7784471/using-sqlite3-on-heroku-cedar-stack 
CANNOT. Ugh. So this is all pointless. Time to migrate to postgreSQL or something that supports sqlite3.

Sigh. I guess I’ll just submit a documentation issue about this whole fiasco.
https://github.com/mitsuhiko/flask/issues/949

Okay, looking into how other people deployed their flaskr projects.
https://github.com/mjhea0/flaskr-tdd
This person has a LOT of documentation and talks about jquery / AJAX. Todo for the future!

https://github.com/mjhea0/flaskr-tdd/blob/master/app.py
WAIT. WTF. It looks like you can use sqlite3, I’m just dumb and you have to include the databse in your root directly, not off in /tmp/ somewhere.

So, looks like DATABASE = ‘flaskr.db’

sqlite3 ./flaskr.db < schema.sql

Note that the way I have it set up — foreman locally will run with Debug = False, heroku will run with Debug = false, and python locally will run with Debug = True. A rather peculiar state of affairs, but fine for me.

Let’s debug the download-from-github install for the Readme.
$ virtualenv –distribute –no-site-packages venv
The –no-site-packages flag is deprecated; it is now the default behavior.
New python executable in env/bin/python
Installing distribute………………………………………………………………………………………………………………………………………………………………………done.
Installing pip…
  Error [Errno 2] No such file or directory while executing command /home/nrw/projects/t…env/bin/easy_install /usr/share/python-vi…p-1.1.debian1.tar.gz
…Installing pip…done.

Installing existing pip-1.1.debian1.tar.gz distribution: /usr/share/python-virtualenv/pip-1.1.debian1.tar.gz
fails. 
Nope, I’m using –distribute and it fails.

 http://stackoverflow.com/questions/10068388/virtualenv-returning-a-no-such-file-or-directory-error
sudo apt-get remove python-virtualenv
sudo pip install virtualenv

bash: /usr/bin/virtualenv: No such file or directory

Sigh.
sudo pip uninstall virtualenv
sudo apt-get install python-virtualenv
NOPE still get the error.
Spaces?
UGH. It’s because my path had spaces.

donation time!

I almost never give to people on the streets. I think my logic is very flawed (irrational fear I’m being taken for naive idiot (does my chinese-american culture play into this?)), but it’s habit by now. I will redress my beliefs at some point in the near future with more research.

In the meantime as I’ve graduated college and started to become an adult, it’s time to consider contributing to organizations I believe in. It’s also true I shouldn’t be thinking about this just once a year, but I think I’ll take this step first and then worry about how morally correct or hypocritical all my actions are.

In part this self-reflection is triggered by this really funny and sobering TED talk:
“In one of the studies, we bring in rich and poor members of the community into the lab and give each of them the equivalent of 10 dollars. We told the participants that they could keep these 10 dollars for themselves, or they could share a portion of it, if they wanted to, with a stranger who is totally anonymous. They’ll never meet that stranger and the stranger will never meet them. And we just monitor how much people give. Individuals who made 25,000 sometimes under 15,000 dollars a year, gave 44 percent more of their money to the stranger than did individuals making 150,000 or 200,000 dollars a year.”

To that end:
(some research that I then ignore)

I live in Boston/Somerville:

These issues affect people I know:

That’s it for today. Man, it sort of sucks that for all of these online donations at least 3% of it goes to credit card companies. Oh well, price of living.
 
Future research:
http://blog.ted.com/2013/05/20/20-resources-for-better-giving-and-living-a-more-altruistic-life/
Peter Singer is pretty controversial in that he proposes that “Effective altruism begins with reason – the realization that all lives are of equal value — and looking for charities that affect the most lives, the most effectively.” People (myself included) always get squeamish when we start applying economics to giving.

He started this site where people pledge to give 10% of their income:
http://www.givingwhatwecan.org/

with some more (interesting!) philosophizing here, where he answers thoughtful questions by readers:
http://www.nytimes.com/2006/12/24/magazine/24singerqa.html?pagewanted=all&_r=0

Using Arduino with Janus Vim (conclusion, who knows how janus/pathogen work, just follow two-year old instructions)

Some time ago I set up my vim config. This is usually a full-day ordeal for some reason I don’t remember. I don’t recall exactly what I did, but everything works now, and I am very happy with how it functions. My previous experiences have left me very wary of “fixing anything that’s not broken” in vim… but enh I want Arduino syntax highlighting in Vim now that there’s the sweeeeet
File > Preferences > Use External Editor 
option in Arduino.

I think for the most part I just installed Janus:
https://github.com/carlhuda/janus

and it has really great autosuggest that makes typing repetitive Arduino code 10x faster.

Lower left window: Autosuggest! So great~

Okay, so how to install Arduino syntax highlighting in Vim Janus setup?
There’s this way that is pretty straightforward:
https://github.com/vim-scripts/Arduino-syntax-file

Just download file into a directory (in my case, ~/.vim/bundle) and add a line or two to .bashrc

nrw@nrw-PC:$ cd ~/.vim/bundle
nrw@nrw-PC:~/.vim/bundle$ git clone https://github.com/vim-scripts/Arduino-syntax-file.git

Edit ~/.vimrc and append two lines:
autocmd! BufNewFile,BufRead *.pde setlocal ft=arduino                         
autocmd! BufNewFile,BufRead *.ino setlocal ft=arduino 

Voila! Done.

===
Below is a recounting of going down the rabbit-hole, if you ever wonder how I can spend a day on my vim config…
===
However, I feel like this isn’t the cleanest way. So how does Vim/Janus handle syntax highlighting?

I feel like somewhere there is a folder of syntaxes that it handles.
Ah! using “locate .vim” I see that the folder is:
/usr/share/vim/vim73/syntax/

Should I just copy it there? Seems incorrect, I think I have to tell Janus somewhere to look for *.pde files.

Based on the pathogen documentation
https://github.com/tpope/vim-pathogen

It appears that I should git clone my .vim file to
~/.vim/bundle

cd ~/.vim/bundle
git clone https://github.com/vim-scripts/Arduino-syntax-file.git

Nope! I don’t have syntax highlighting after restarting vim (I double-checked with :scriptnames) . Perhaps install in .janus folder instead?
 cp ~/.vim/bundle/Arduino-syntax-file/ ~/.janus -r

Nerp still no syntax highlighting ;__; where do I tell it to look for both .ino and .pde files?

Hmm, it also appears that my Janus distribution is out-of-date based on the documentation (there is no NERDCommenter in my Janus as far as I can tell). But if I try to update
~/.vim$ rake
/usr/local/bin/rake:9:in `require’: no such file to load — rubygems (LoadError)
    from /usr/local/bin/rake:9

Uh oh. I can feel the rabbit hole widening.
http://stackoverflow.com/questions/2896485/no-such-file-to-load-rubygems-loaderror

nrw@nrw-PC:~/.vim$ which -a ruby
/usr/bin/ruby
/home/nrw/.rvm/bin/ruby

nrw@nrw-PC:~/.vim$ sudo update-alternatives –config ruby
There are 2 choices for the alternative ruby (providing /usr/bin/ruby).

  Selection    Path                Priority   Status
————————————————————
* 0            /usr/bin/ruby1.9.1   51        auto mode
  1            /usr/bin/ruby1.8     50        manual mode
  2            /usr/bin/ruby1.9.1   51        manual mode

Yep. Okay, so I need to nuke one of my versions.
 sudo apt-get remove ruby

which -a ruby
/usr/bin/ruby
/home/nrw/.rvm/bin/ruby

http://learnvimscriptthehardway.stevelosh.com/chapters/43.html

Hmm, still two versions. Well. *shrug* Let’s just kill one version. Ugh. I bet this is leftovers from all the EdX stuff I did that never worked and I didn’t really document because I was so frustrated.
AHHHH this is why I sometimes just NUCLEAR and reinstall my OS. Eheh. ^^;

Nooo nooo bad brain nooo it is 10 pm on a Wednesday night. It is NOT time to reinstall my OS. 

Okay. Screw not messing up my system, it’s already screwed up.
1) Add execute pathogen#infect()     to my .vimrc

NOPE
Error detected while processing /home/nrw/.vimrc:
line  134:
E117: Unknown function: pathogen#infect
E15: Invalid expression: pathogen#infect()

Okay… Maybe I’ll just follow directions.

install details
Just put the file in your vimfiles/syntax folder. To automatically use it on PDE files, add the following line to your vimrc file (or e.g filetype.vim to install it for all users) :

autocmd! BufNewFile,BufRead *.pde setlocal ft=arduino

Okay…. I don’t have a /syntax folder. What is going on? Uhh.

Well, I just put it into ~/.vim/bundle. And it works. ~__~ ah well. I guess I just have to follow instructions…

===
Colorize bash prompt
https://help.ubuntu.com/community/CustomizingBashPrompt

so now my ~/.bashrc just contains: 
color_prompt=yes                                                              
                                                                              
if [ “$color_prompt” = yes ]; then                                            
    PS1=’${debian_chroot:+($debian_chroot)}[33[01;32m]u@h[33[00m]:[>
else                                                                          
    PS1=’${debian_chroot:+($debian_chroot)}u@h:w$ ‘                       
fi

For further reading:
http://www.reddit.com/r/programming/comments/697cu/bash_users_what_do_you_have_for_your_ps1/

===
Also, vim things I always forget:
To toggle whitespace,
:set list
:set nolist

To toggle linenumbers,
:set nu
:set nonu

To toggle paste-mode,
:set paste
:set nopaste

Leader key
stackoverflow.com/questions/1764263/what-is-the-leader-in-a-vimrc-file
let mapleader=”,”

So “,n” will open NerdTree.