Monday, January 27, 2014

AT week-long hike planning (PONIES!) || MIT Outdoors Club, Winter School: Easy Hike


Grayson Highlands State Park, aka PONIES
Out of the blue one day a few weeks ago, my friend Judy asked me if I wanted to backpack a week-long section of the Appalachian Trail with her in late May.

I didn't grow up doing sports, exercise, or any sort of physical activity, let alone outdoors activities. Only with a more varied social circle in college (and in particular with more time and money post-undergrad) have I started to take more interest in outdoors activities or physical exercise at all.

Our current preferred choice is to hike SOBO (southbound) from Atkins to Damascus at a leisurely place, taking time to enjoy the PONIES. :D

http://www.whiteblaze.net/forum/archive/index.php/t-39258.html
We hiked Southbound from Atkins to Damsacus (75 Miles) and it seemed to have some level and gradual climbing. We did it in 5 days but you could make it six days the Grayson Highlands are great.
One piece of advice is as you approach Grayson Highlands Park and the area around Mt. Rogers, I'd pack an extra day's worth of food, as this is a place you'll won't want to hurry thru, and may well want to slow down, and maybe even take a full day off. "Zero" days on the Trail can be better than days off in town, but very few folks ever do this, partly because they haven't brought along enough provisions.
I learned from experience that it's far preferable to shuttle to your starting point and get back when you get back, than to hike to a prearranged shuttle at a set time and have to worry about making it on time for your whole hike.
We are worried about how we will get there and back. It would suck to rent a car for a week and just have it sit at the trailhead burning rental money. We'll see. Hiking the AT in NY would be much easier, since we can reach the trail by public transit, but if we're committing a week's worth of time on the trail and far more preparing for the hike, we will probably prefer to hike a prettier section of the trail.

In any case, we are starting several months ahead of time because of our lack of experience with backpacking. Areas of preparation include food, gear, physical fitness, and backpacking experience.


My initial plan was to climb the porter square T stop stairs (before the pay entrance) every day, since it only takes around 45 minutes including walking there and back from my house (because I can only climb 15 minutes of stairs).

picture proof I sent to Judy, who lives in NY
The stairs are really narrow so it's a good thing I'm the only one doing this. However, this plan fell through, as most of my plans tend to do, in part because I have been feeling crunched for time between NarwhalEdu's contract work and our kickstarter work. The new plan is to resume when we finish our MIT Office of Digital Learning contract work, probably in mid-February. It's interesting. I never found much point in exercising even though it's definitively good for you, but I find the idea of exercise a lot less tedious and more interesting when there's a goal I am seriously interested in in mind (spending time on an adventure with a great friend).

In the meantime, on the weekends, to help convince myself to exercise, I have been participating in MIT Outdoors Club (MITOC) winter school (open MIT students, affiliates, alums, and the general community). "Participating," since for various reasons this past weekend was my first winter hike to Noanet Woods in Dover, MA (about 40 minutes drive from MIT).

Note: In case they send out the email with links four hours after sign-ups open again next year and screw over newcomers on the first weekend trips, go to http://web.mit.edu/mitoc/www/#join_trip to sign up for a trip. Trips fill up fast so be sure to sign up exactly Wednesday at noon. Getting off the waitlist is possible if  you're high up, so you can try showing up to the pre-trip meeting even if you are waitlisted.

we are super over-prepared, since it is a learning exercise for winter school beginners and due to winter school safety guidelines
I chose the easiest hike possible, since I know I'm not really in shape despite biking on average 30 minutes a day (I bike slowly), and I don't have experience hiking, so I don't know if 7 miles is too much for me or not. I don't want to drag my group down.



I still prepared for this hike quite a bit, which helped make it a mostly comfortable and enjoyable experience instead of one where I was freezing my extremities off. Biking in sub-zero weather forced me realize I need to invest in proper winter clothing too.

Over the last few weeks,
  1. I bought Neff Women's Digger Gloves (black) off of SteepandCheap, since I thought they were skiing gloves that they would be fine. They were such fail I returned them. Maybe they require liners underneath for people with cold hands like me :/
  2. I went to Harbor Freight in Medford, MA and discovered Ocean State Job Lot next to it, where I bought lots of non-cotton items: liner socks for $2.50, wool blend tall socks and boot padding socks for $3 to $5 each, a pair of thinsulate 40g gloves for $5, and fleece leggings for $5
  3. I went to Target in Somerville, MA and discovered CWPrice, where I bought a base layer (pants+shirt) for $10
  4. I dug out my orange sweater and grandma's windbreaker (:/ she passed away a few years ago)
  5. I learned from Cappie to tie cloth or a scarf around my nose and mouth to keep my face warm. This tactic does tend to fog up my glasses. I need to remember to wear contacts, although I've gotten better about breathing with my mouth in overbite formation so that the hot air is redirected through the bottom
  6. Jordan from MITERS lent me a pair of thick Bonfire gloves that seem to be on par or slightly warmer than my thinsulate gloves. He found them lying around for months so he gave them to me for free.
  7. Bought a magnesium fire starter at harbor freight, and a compass and whistle from MITOC
Now, my everyday wear is:
  • 3 leg layers, 3 top layers, 2 layers of socks, a hat, a scarf, and the thinsulate gloves
On the hike in around 20F I wore
  • 3 leg layers (leggings, base layer, and jeans, because I didn't have thick / wind or water-proof non-cotton shell layer. I notified my trip leaders and they brought polyester hiking pants, but I stuck with the jeans because it wasn't precipitating and my legs felt warm just standing outside)
  • 3 body layers (base layer, my awesomely warm patagonia jacket, and a windbreaker) -- no poofy layer was needed
  • Two socks
  • Rain boots with liners (winter hiking boots would have been better, but it is such as hassle to leave a deposit and check in and out equipment, even if the monetary cost is already much better)
My toes were painfully cold for 1/3 of the hike and then suddenly warmed up in such a way that at first I was afraid they'd gone numb. Then they were nice and warm. Perhaps this phenomena was due to cold-induce vasodilation. I wish I didn't have to deal with painfully cold toes at all though, and I asked some of the other hikers and their toes were warm throughout! I've always had problems with cold toes though.

frozen lake
Noanet Peak gave a view of the Boston skyline
It was an interesting experience, hiking on snowy trails. A lot of normal people passed us, running with their dogs or walking on skis.

pawprints!

It was a fun group. We joked about "traversing noanet" and "bagging noanet peak" (387 ft) :)

I look forward to the next hike. I learned that MITOC Winter School doesn't teach you first aid, and some of these were covered in lectures that I missed, but I things I want to learn / hands-on:
  • use a compass (oops I missed this lecture, ssshhh I am supposed to know it before going on a hike)
  • emergency signals
  • start a fire in wet conditions
  • wilderness first aid
  • use a stove
  • other TMI things (there was such a great TMI lecture during the second set of mandatory lectures, covering all the female-specific issues) (many things are different in winter -- you can't dig a hole for your poop, for instance, because the ground is frozen)
I want to go on an overnight winter trip too, but they all tend to be for intermediate hikes and up. I suppose there's next year, and I can first practice with easier spring weekend trips.

Tuesday, January 14, 2014

WTFisThisRegister, Beginner's Flask App: Dictionary based on Flaskr (uses sqlite3) and deployed to Heroku

Over the course of a night and morning I built a small flask app (Flask is a micro web framework for Python. Django is similar but more heavy-duty). It is essentially a dictionary. You can search for an entry by its keyword, or view the helptexts for all the entries, or (if you are logged in) add and remove entries. It is a straightforward derivative of excellent the Flaskr tutorial.



For me, I am really proud of this app even though it is dirt simple and not much to look at because this is the first time I really interacted with a database in a programming language.

The intent was to make a database of documentation for the AVR Atmega328P microcontroller. However, since then I have decided to move to using a mediawiki (sample page).

The app may be seen live at salty-retreat-5363.herokuapp.com. The username is "admin" and the password is "default". The database gets wiped at least once every 24 hours.

If you'd like to run the app locally and play around with source code, please see https://github.com/nouyang/WTFisThisRegister for detailed instructions. That page also details how to deploy this app (or similar ones, such as Flaskr) to Heroku. I used http://dillinger.io/ to play around with markdown online and learn it and actually document this project with a proper README.


Things Learned


Here are some things I learned in the course of making this:
  1. Gitignore files need to be created before you run "git add ." -- in general, the line telling it to ignore a file needs to exist before you added it to the version control, git won't automagically remove it for you.
  2. Heroku DOES play with sqlite3 but not happily, for instance the database will be wiped at least once every 24 hours. See https://devcenter.heroku.com/articles/sqlite3.
  3. Virtualenv does NOT like spaces in your directory path! This was tricky to debug as even in -v verbose mode the full path for my virtualenv packages was shortened and so I couldn't even notice that my path had spaces in it.
  4. Databases use cursors.


The Process

Here are some images from the process. Click on them to view them in full resolution.

1. First I started out just using a dictionary (no database involved) and a single hello.py file.

2. I then followed the Flaskr tutorial. After doing so I wanted to modify it so that the text was monospaced and preserved whitespaces, because it doesn't by default.


I used the "pre" tag at first.


but soon switched to using CSS.
This actually caused me a lot of headache later troubleshooting why there was random whitespace in my design, because I didn't restrict where the CSS was applied enough. So much whitespace:


 In the end I had this file: https://github.com/nouyang/WTFisThisRegister/blob/master/static/style.css
.helptext { preserve whitespace with wrapping CSS }
and the corresponding code in the html file was
 <div class="helptext"%gt;
{{ entry.helptext|safe }}</div>

3. I wanted to be able to delete entries, so I added a delete button.
In show_entries.html oops can't show it because blogger can't escape this HTML properly :/ In the main python file:
@app.route('/delete', methods='POST'])                                                                                         
def delete_entry():                                                            
    if not session.get('logged_in'):                                           
        abort(401)                                                             
    g.db.execute('delete from entries where keyword = ?', [request.form['entry_to_delete']])
    g.db.commit()                                                              
    flash('Entry was successfully removed')                                    
    return redirect(url_for('show_entries')) 
The "?" is a safe why for sqlite3 to accept text (in this case, whatever was entered in the form with the name entry_to_delete).

4. I learned to play around with the database to test my SQL commands in the terminal:
 $ sqlite3 /tmp/flaskr.db
sqlite> select helptext from entries where helptext='PCMSK0'


5. Finally I added a search functionality to the site:


@app.route('/search', methods=['POST'])                                        
def search_entries():                                                          
    keyword = request.form['searchterm']                                       
    cur = g.db.execute('select helptext from entries where keyword = ?',          
                 [keyword])                                                    
    result = [dict(keyword=keyword, helptext=row[0]) for row in cur.fetchall()]
    return render_template('search.html', result=result)                    
And set the homepage to be the search page:

@app.route('/')                                                                
def show_search():                                                             
  return render_template('search.html') 

6. I learned about initializing databases, namely by dumbly thinking I needed to include the flaskr.db file and no schema.sql file in the repository to allow other people to get it working on their systems. Actually, I was just missing a line, "init_db()", before "app_run()". In the end I am still keeping it without that line so that I have persistence in the database (I can stop the server and restart it and the database entries will still be there).

All in all I learned a lot! Do browse through the source code and play it with yourself if you'd like.



 =========================================

Liveblog Notes (kept here for reference)


http://flask.pocoo.org/docs/quickstart

http://maximebf.com/blog/2012/10/building-websites-in-python-with-flask/
https://github.com/trtg/flask_assets_tutorial

(did not use: http://www.realpython.com/blog/python/python-web-applications-with-flask-part-ii-app-creation/)

http://flask.pocoo.org/docs/tutorial/introduction/ Ah! Here's the hands-on quickstart.


Hmm, well now the newlines show up but long lines aren't wrapped.
To fix:
http://stackoverflow.com/questions/642140/retain-newlines-in-html-but-wrap-text-possible

.page { width: 70%;
min-width:35em;
border-radius:5px;
}

Modifying the database name: How are you supposed to properly do this? I just deleted /tmp/flaskr.db and then ran in the shell:
sqlite3 /tmp/flaskr.db < schema.sql
and things seemed to work again.

Include a delete button in the future:

If you don't want your data wiped clean, DO NOT do 
$ python
>>> from flaskr import init_db
>>> init_db()

Instead, to play with sqlite3 commands,
sqlite3 /tmp/flaskr.db

And then use the same commands as in flask, but with semicolons.
So to lookup something we need the WHERE command:
e.g. for me it is:
select helptext from entries where keyword=''PCMSK0";

Uhm, okay, now how to display it in my flask app?

ahh what are database i am so confused
http://www.zetcode.com/db/sqlitepythontutorial/
http://docs.python.org/2/library/sqlite3.html

Let's install the AVR eclipse plugin
http://avr-eclipse.sourceforge.net/wiki/index.php/The_AVR_GCC_Toolchain#Debian_and_Ubuntu
http://www.instructables.com/id/How-to-get-started-with-Eclipse-and-AVR/step13/Quick-Tour-Of-Cool-Features/ (windows)

sudo apt-get install eclipse (270+ MB! sigh)
sudo apt-get install gcc-avr binutils-avr gdb-avr avr-libc avrdude

http://avr-eclipse.sourceforge.net/wiki/index.php/Plugin_Download

Heroku deploy
http://flask.pocoo.org/docs/quickstart/#quickstart-deployment
https://devcenter.heroku.com/articles/getting-started-with-python

  1. Install heroku toolchain
  2. Login to heroku
  3. Install and activate virtual environment
  4. Install dependencies (why do I need gunicorn? not sure) 
  5. Make a Procfile
  6. Run foreman and check in browser that app is up on localhost
  7. Download the python .gitignore and add the "venv" line to the top
  8. my own step: add to github, git remote add origin https://github.com/nouyang/WTFisThisRegister.git

https://help.github.com/articles/ignoring-files
OH. The .gitignore file does NOT go in the root directory directly. No wonder everything failed and I just had to delete my github repo and also the .git folder in my root directory.

Instead,
For example, you might create the file at ~/.gitignore_global and add some rules to it. To add this file to your cross-repository configuration, run git config --global core.excludesfile ~/.gitignore_global. Local per-repository rules can be added to the .git/info/exclude file in your repository. This method can be used for locally-generated files that you don't expect other users to generate.

Wait, but what if there's one file, e.g. a passwords for the database file, that I don't want to commit? Maybe it's just messed up because I did the git init and then added the gitignore file or something funky, I don't remember. http://stackoverflow.com/questions/1139762/gitignore-file-not-ignoring

Okay, so now I want to pull the config parameters (passwords, admin usernames, secretkeys) out into a separate file that is gitignored.
Done. Created a file called databaseconfig.py, added the line "from databaseconfig import secretkey, username, password" to the main python file.


Oh I should generate a real secret key
http://stackoverflow.com/questions/18247971/print-python-os-urandom-output-on-terminal
python
>>> import os, binascii
>>> binascii.hexlify(os.urandom(24))

Whoa they actually mean create the file ~/.gitignore_global, not create ~/.gitignore_global/.gitignore
Whoops.

http://stackoverflow.com/questions/4824188/git-ignore-vim-temporary-files
Also, vim and gedit create swap files not covered by github's gitignores, so add
*~
and
*.swp
*.swo

to ~/.gitignore_global

Yes! Succesfully used .gitignore for once.

(venv)nrw@nrw-PC:~/projects/WTFisThisRegister$ git commit -m "init"
[master (root-commit) 35b1400] init
 10 files changed, 228 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 Procfile
 create mode 100644 README.md
 create mode 100644 WTFisThisRegister.py
 create mode 100644 requirements.txt
 create mode 100644 static/style.css
 create mode 100644 templates/layout.html
 create mode 100644 templates/login.html
 create mode 100644 templates/search.html
 create mode 100644 templates/show_entries.html
Now add it to github.
git remote add origin https://github.com/nouyang/WTFisThisRegister.git
git push -u origin master


Okay, now to finish the heroku deploy.
heroku create 
git push heroku master
Permission denied (publickey).
fatal: The remote end hung up unexpectedly

Whoops I need to add this computer to heroku.
https://devcenter.heroku.com/articles/keys
$ ssh-keygen -t rsa
/home/nrw/.ssh/id_rsa already exists.

Whoops, let's not overwrite that, maybe it will break github. 
cat ~/.ssh/id_rsa.pub

and copy that into heroku under Account (https://dashboard.heroku.com/account).

Alright, let's try pushing to heroku again.
-----> Launching... done, v3
       http://salty-retreat-5363.herokuapp.com deployed to Heroku

Yay!!

Oh noes. It doesn't work on heroku... Is that because I gitignored the databaseconfig.py file and so it doesn't exist on heroku?


Guhuhuh I guess I should learn to use github branches.

Okay nevermind, I found a better answer than having two branches to maintain and update if I ever change anything (bletch).

This gives a clear explanation and example: https://devcenter.heroku.com/articles/config-vars

So in the main python file I put
SECRET_KEY = ENV['SECRETKEY']
and then in the terminal I type
heroku config:set SECRETKEY=blahblah.

Then, git add, git commit, and then git push heroku master.

NOPE that didn't fix it ;____; wahh I am sad.

Let's research....

http://stackoverflow.com/questions/18037027/flask-app-failing-on-heroku-but-working-with-foreman
WHOA good point I forgot to turn debug off! Whaaaa. Okay that is terrible of me.
# configuration
DEBUG = False

WAH. still sad.

Let's try to fix foreman by adding the var to .env... Does that work?

.env does not like spaces around the equal sign assignment operator!
S3_KEY=mykey
NOT 
S3_KEY = mykey

Oh! Make sure to add "import os" to the main python file if we are using os.environ. DUH.
Note: Only today I realized that in pasting URLs from the location bar in chromium-browser (ubuntu 12.10) into blogger, the URLs are automatically turned into hyperlinks, but they remain plain old text and I have to manually turn them into hyperlinks when I copy them from firefox, my main browser :( Such a weird specific bug.

Ah? It still doesn't work?? Why? It works locally on foreman, but not remote on heroku.

Okay, let's look at the heroku logs.
https://devcenter.heroku.com/articles/logging
ImportError: No module named databaseconfig

Whoops. So a gitignore issue.

UGH. Everything becomes 10x more complicated when working in a virtual environment. Fine. I'll take the hit and jump full out into non-localhost environment, where I can't just run "python blah.py" I have to run "foreman start" and other issues. Because foreman looks for a ".env" file. But python does not. (http://stackoverflow.com/questions/12335488/cannot-use-environment-variables-for-settings-in-django)

The solution is to use os.environ.get() instead.

Okay, so I'll just write a short if statement checking if os.environ.get() is None. 

SECRET_KEY = os.environ.get('SECRETKEY') #for heroku deploys.                
if SECRET_KEY == None:                                                      
    SECRET_KEY = secretkey    
Great, now my config works in with both "foreman start" and "python WTFisThisRegister.py".

Now to get databaseconfig to show up in heroku. Do I want to?
No. A cleaner solution instead of all this is to use try, importerrors. Since the present of databaseconfig.py can be used to indicate that we are running it locally.


ahhhh now the /entries works locally but there is an internal server error remotely. WHY.
http://stackoverflow.com/questions/8950674/debugging-a-flask-app-running-in-gunicorn
Instead of foreman start, just use foreman run python app.py if you want to debug your application in development.

 Maybe it's an issue with gunicorn or procfile or whatever that stuff is?
 web: gunicorn WTFisThisRegister:app        
http://ryaneshea.com/lightweight-python-apps-with-flask-twitter-bootstrap-and-heroku
changed to
web: python WTFisThisRegister.py

Okay, still works locally but not remotely. Maybe it is a port issue. Sucks that these tiny changes debugging production (heroku environment) take forever to try (30 seconds for each time I git push heroku master)

2014-01-12T05:58:48.956357+00:00 heroku[web.1]: State changed from crashed to starting
Yay! Fixed ports issue.

AGH. Still internal server error. WTF.
http://stackoverflow.com/questions/7653010/import-sqlite3-with-python2-7-on-heroku
So sqlite3 could NOT be used on the previous version of heroku. However, I think it should be able to now?
https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem
http://stackoverflow.com/questions/7784471/using-sqlite3-on-heroku-cedar-stack 
CANNOT. Ugh. So this is all pointless. Time to migrate to postgreSQL or something that supports sqlite3.

Sigh. I guess I'll just submit a documentation issue about this whole fiasco.
https://github.com/mitsuhiko/flask/issues/949

Okay, looking into how other people deployed their flaskr projects.
https://github.com/mjhea0/flaskr-tdd
This person has a LOT of documentation and talks about jquery / AJAX. Todo for the future!

https://github.com/mjhea0/flaskr-tdd/blob/master/app.py
WAIT. WTF. It looks like you can use sqlite3, I'm just dumb and you have to include the databse in your root directly, not off in /tmp/ somewhere.

So, looks like DATABASE = 'flaskr.db'
sqlite3 ./flaskr.db < schema.sql

Note that the way I have it set up -- foreman locally will run with Debug = False, heroku will run with Debug = false, and python locally will run with Debug = True. A rather peculiar state of affairs, but fine for me.


Let's debug the download-from-github install for the Readme.
$ virtualenv --distribute --no-site-packages venv
The --no-site-packages flag is deprecated; it is now the default behavior.
New python executable in env/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...
  Error [Errno 2] No such file or directory while executing command /home/nrw/projects/t...env/bin/easy_install /usr/share/python-vi...p-1.1.debian1.tar.gz
...Installing pip...done.

Installing existing pip-1.1.debian1.tar.gz distribution: /usr/share/python-virtualenv/pip-1.1.debian1.tar.gz
fails. 

Nope, I'm using --distribute and it fails.


 http://stackoverflow.com/questions/10068388/virtualenv-returning-a-no-such-file-or-directory-error
sudo apt-get remove python-virtualenv
sudo pip install virtualenv

bash: /usr/bin/virtualenv: No such file or directory

Sigh.
sudo pip uninstall virtualenv
sudo apt-get install python-virtualenv

NOPE still get the error.
Spaces?

UGH. It's because my path had spaces.