No intro to profiling in Python is complete without any mention of line_profiler (http://pythonhosted.org/line_profiler/). As the name suggests, it gives you line-by-line profiling information. In a scripting language such as Python, I find this much more useful.
I discovered line_profiler last week when I was trying to learn how to profile in Python. You show it your code, and it shows you what's slow - this is exactly what a profiler should be.
I really want to start a campaign to get something similar built for JavaScript. JavaScript has the most diverse user base (from people who barely know how to use their computers all the way up to Googlers), yet our profiling tools can be intimidating/hard-to-decipher even for people who are quite comfortable with the rest of the language.
My mental model for my code is my code. Showing me a data-grid with a tree of function calls (many of which may be in external libraries I don't control) isn't nearly as helpful as showing me, line-by-line, how fast my code runs. (This is especially true since not every line that takes time to run makes a function call.)
Definitely useful, I'll revisit the slides later when I'm looking at optimizing.
I'm working through some of CPU overhead issues with Django now and wishing occasionally that I had chosen to use Sqlalchemy as my primary database tool. Django seemed like the easier solution at the time because of the steep learning curve of Sqlalchemy. I think it's worth learning, especially now that I've hit some edges in Django's ORM. I suppose it's good to know I can always use a crazy hybrid of the two libraries if needed.
I got real sad that a presentation in 2013 is being written against Python2 syntax still. I do admit I still have to go out of my way sometimes to find a complete p3 stack, but I've never failed to find the tools I needed in versions supporting 3 somewhere.
Unfortunately there are still a lot of bindings and third party libraries that need python2. Recently I've been playing with python-alsaaudio and python-cwiid on linux, and they both seem to be require python 2.
It seems like it's not going to go away any time soon. I'd be surprised if most new work was even being done in python 3, even though it's been out for almost 5 years now.
I'm hoping that now that we have the major framework - Django supporting python 3 in Alpha (now) and fully (in v1.6) we'll finally see a slow move to python 3.
What I wound up doing in a case like this was dump (using copy) into a temp table, then run a sql function on the data. That way, you get it done in as few round trips as possible, essentially 3 statements (create temp table, copy, select dostuff()). The database is real fast about doing what it needs to do when you give it one big block of stuff to work with.
And, if you've got a serial column, you're going go have a primary key/uniqueness constraint on it. Retrying that when it fails at interesting times is going to cost far more time and brain cells than letting the db do it.
I think it is interesting to have it in there for academic purposes, to see how fast you could go sticking with the high level ORM API. Once he drops down into the expression language, he goes back to generating the ids server side.
