{"id":7898,"date":"2015-11-07T04:57:04","date_gmt":"2015-11-07T04:57:04","guid":{"rendered":"https:\/\/unknownerror.org\/index.php\/2015\/11\/07\/memory-error-when-using-pandas-read_csv-open-source-projects-pydata-pandas\/"},"modified":"2015-11-07T04:57:04","modified_gmt":"2015-11-07T04:57:04","slug":"memory-error-when-using-pandas-read_csv-open-source-projects-pydata-pandas","status":"publish","type":"post","link":"https:\/\/unknownerror.org\/index.php\/2015\/11\/07\/memory-error-when-using-pandas-read_csv-open-source-projects-pydata-pandas\/","title":{"rendered":"Memory error when using pandas read_csv-open source projects pydata\/pandas"},"content":{"rendered":"<p>I am trying to do something fairly simple, reading a large csv file into a pandas dataframe.<\/p>\n<p>This is what I am using to do this:<\/p>\n<pre><code>data = pandas.read_csv(filepath, header = 0, sep = DELIMITER,skiprows = 2)\n<\/code><\/pre>\n<p>The code is behaving quite erratically. It either fails with a memory error (detailed error message as P.S.), or just never finishes (Mem usage in the task manager stopped at 506 Mb and after 5 minutes of no change and no CPU activity in the process I stopped it).<\/p>\n<p>I am using pandas version 0.11.0. I am aware that there used to be a memory problem with the file parser, but according to <strong>http:\/\/wesmckinney.com\/blog\/?p=543<\/strong> this should have been fixed. The file I am trying to read is 366 Mb, the code above works if I cut the file down to something short (25 Mb). It has also happened that I get a pop up telling me that it can&#8217;t write to address 0x1e0baf93&#8230;<\/p>\n<p>I am running the code in debug in Visual Studio, using Anaconda and <strong>PTVS<\/strong> (the step-by-step debug, F5).<\/p>\n<p>A bit of background &#8211; I am trying to convince people that Python can do the same as R. For this I am trying to replicate an R script that does<\/p>\n<pre><code>data<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>I am trying to do something fairly simple, reading a large csv file into a pandas dataframe. This is what I am using to do this: data = pandas.read_csv(filepath, header = 0, sep = DELIMITER,skiprows = 2) The code is behaving quite erratically. It either fails with a memory error (detailed error message as P.S.), [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-7898","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/unknownerror.org\/index.php\/wp-json\/wp\/v2\/posts\/7898","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/unknownerror.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/unknownerror.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/unknownerror.org\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/unknownerror.org\/index.php\/wp-json\/wp\/v2\/comments?post=7898"}],"version-history":[{"count":0,"href":"https:\/\/unknownerror.org\/index.php\/wp-json\/wp\/v2\/posts\/7898\/revisions"}],"wp:attachment":[{"href":"https:\/\/unknownerror.org\/index.php\/wp-json\/wp\/v2\/media?parent=7898"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/unknownerror.org\/index.php\/wp-json\/wp\/v2\/categories?post=7898"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/unknownerror.org\/index.php\/wp-json\/wp\/v2\/tags?post=7898"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}