About Archive Tag Cloud Translations RSS

You are writing a comment about Huge CSV and XML Files in Python, here is a quick summary:

Quick walkthrough of my code for converting a very large CSV file into a very large XML file using the Python standard libraries. Despite a few issues along the way, was a very pleasant experience.


You are responding to this comment written by Stu on January 23rd 2009, 07:22.

Throughput is quite good - I'm processing 29mb csv and inputting to a database via django. It takes about 30minutes for 200k+ records (there are a few joined tables on these too).

My app isn't time critical, but it does feel like it could be quicker*.

  • I guess I could generate raw sql, but then the nice ORMness is lost.


Please be aware that comment forms go stale after one hour.





Comments may make use of LifeFlow MarkDown. Raw html will be escaped.


Quick Introduction to LifeFlow MarkDown Syntax

A highlighted code block:

@@ ruby
def a (b, c):
  b * c
end
@@

Other common languages work as well: scheme, python, java, html, etc.

Other markdown syntax:

 ### This is an h3 title
#### This is an h4 title
**this is bold**
*this is italics*

1. This is an
2. ordered list

* And an unordered
* list too

[this is a link](http://www.lethain.com/ "Lethain")