EuroPython 2014 – Python under the hood

This is actually a summary of few talks which deals with “under the hood” topics, the talks partially overlap. Those topics include – memory allocation and management, inheritance, over-ridden built-in methods, etc.

Relevant talks –
The magic of attribute access by Petr Viktorin
Performance Python for Numerical Algorithms by Yves
Metaprogramming, from Decorators to Macros by Andrea Crotti
Everything you always wanted to know about Memory in Python but were afraid to ask by Piotr Przymus
Practical summary –
  • __slots__ argument – limited the memory allocation for objects in Python by overriding the __dict__ attribute.
  • Strings – the empty strings and strings of length=1 are saved as constants. Use intern (or sys.intern in python 3.x) on strings to avoid allocating string variables with the same values this will help making memory usage more efficiency and quicker string comparison. More about this topic here .
  • Numerical algorithms – the way a 2-dimensional array is allocated (row wise or column wise) and store has a great impact on the performance of different algorithms (even for simple sum function).
  • Working with the GPU – there are packages that process some of the data on the GPU. It is efficient when the data is big and less efficient when the data is small since copying all the data to the GPU has some overhead.
  • Cython use c “malloc” function for re-allocating space when list \ dictionaries \ set grow or shrink. On one hand this function can be overridden, on the other hand one can try to avoid costly processes which cause space allocation operations or to use more efficient data structures, e.g list instead of dictionary where it is possible.
  • Note the garbage collector! Python garbage collector is based on reference count. Over-ridding the __dell__ function may disrupt the garbage collector.
  • Suggested Profiling and monitoring tools – psutilmemory_profilerobjgraphRunSnakeRun + Meliaevalgrind
Bottom line of all of those – “knowledge itself is power”. I.e. knowing the internal and the impact of what we are doing can bring to a significant improvements.
There are always several ways to do things and each has cons and pros fitted to the specific case. Some of those are simple to implement and use and can donate to a great improvement on both running time and memory usage. On the other hand some of those suggestion are really “shoot in the foot” – causing memory leaks and other unexpected behavior, beware.

Graph Databases, a little connected tour

EuroPython talk by: Francisco Fernández Castaño
 
 
The talk was classified as novice and so it was – very basic graph database ideas.
 
Castaño started by presenting the general idea of graph databases and showing some use cases.
 
One of the most known use cases – social media data, friends-of-friends.
 
Then he presented Neo4j which is graph database written in Java but originally was written in Python. Neo4j is known to be very scalable and to support ACID transactions. Another nice property about Neo4j is the ability to extend the give rest API but your own needs.
 
The next part of talked focused on the cypher language – which is the way to query the Neo4j database. Neo4j have some nice UI properties and I really missed some reference or example for that on this talk.
 
Last but not least tip given in the talk, you can try Neo4j without having to install it on your machine – http://www.graphenedb.com/ (free sand box of up to 1k nodes and 10k edges).

Log everything with logstash and elasticsearch

EuroPython talk by: Peter Hoffmann@peterhoffmann

 
A talk by Peter Hoffmann from Blue Yonder which is one of the sponsors of the conference.

 

Another very good talk by an experienced speaker. However the name  is kind of misleading. Yes – logstash and elasticsearch were mentioned however to main concept of the talk was really logging chain and centralized logging while logstash and elasticsearch are two tools along this chain and there are some alternatives in every step of this chain.

This talk gave me a lot to think about the logging on the company aspect and how should they run, monitor, etc. Also there is some tension to solve \ define about what is logged and what error message \ outputs an app should provide (“logging best practices”).

The video is already available in the EuroPython site and I hope that the slides would be available too soon.

Full Stack Python

EuroPython talk by: Matt Makai, @mattmakai 

 
I expected this talk to be full with buzzwords and it was… but in a good sense. 

 
Makai is the builder of Full Stack Python site as such he spoke about what you need from the moment you have an idea to a python web app until you deploy including all the essential steps – wsgi server, hosting, logging etc.
 
Every layer in the site include relevant link and tutorials. Good starting point for a python web-app developer.
 
The talk was interesting due to Makai enthusiasm to teach and to share his knowledge (and of course to promote his site and that’s legit as well) and his professional knowledge. There are really few people the can speak so fluently for 25 minutes.