When I run my script locally there are not issues, but when I run it in pythonanywhere it returns this error: TypeError: argument 1 must have a "write" method
Why is this?
How can I share my script here for viewing?
Forums
TypeError: argument 1 must have a "write" method python
You can share a script here. Indent it with 4 spaces for code blocks. But yes I would need to see the code and understand how you were trying to run it before I could answer that question.
hansel
|
459
posts
|
PythonAnywhere staff
|
Thanks,
The line that is returning the error is the following
writer = csv.writer(file)
Sounds like the file object isn't an actual file... can you share more code?
Here is a chunk that includes the preceding file object, which includes the directory.
file = urlopen('https://www.pythonanywhere.com/user/mcca2174/files/home/mcca2174/Dropbox/pyanywhere/Python_E/scrape_results.csv', 'wb')
writer = csv.writer(file)
what is the output when you do the following:
print dir(file)
thanks
rcs1000
|
311
posts
|
['__doc__', '__init__', '__iter__', '__module__', '__repr__', 'close', 'code', 'fileno', 'fp', 'getcode', 'geturl', 'headers', 'info', 'msg', 'next', 'read', 'readline', 'readlines', 'url']
Hmmm: that definitely looks like a file object.
But it looks like read-only file object - there is a read method, but not a write one.
Ah ha.
I've worked it out. You're using urlopen to open the file. You should just be using open.
So, try this:
file = open('home/mcca2174/files/home/mcca2174/Dropbox/pyanywhere/Python_E/scrape_results.csv', 'wb')
That fixed the initial issue. However, it returned this error:
Traceback (most recent call last):
File "/home/mcca2174/opencsvtest.py", line 12, in <module>
file = open('https://www.pythonanywhere.com/user/mcca2174/files/home/mcca2174/Dropbox/pyanywhere/Python_E/scrape_results.csv', 'wb')
IOError: [Errno 2] No such file or directory: 'https://www.pythonanywhere.com/user/mcca2174/files/home/mcca2174/Dropbox/pyanywhere/Python_E/scrape_results.csv'
>>>
Is there a particular method for establishing a directory through my dropbox from within pythonanywhere?
[edit by admin -- tidied traceback for legibility]
To open a file with open you should just use the filename, not a URL. So your code should say
open('/home/mcca2174/Dropbox/pyanywhere/Python_E/scrape_results.csv', 'wb')
--- that is, remove the https://www.pythonanywhere.com/user/mcca2174/files from the start of the filename.
In general, if you want to access a file from your PythonAnywhere Dropbox share, just put /home/mcca2174/Dropbox at the start.
Having trouble with the syntax, give me a second
Here's my script
import urllib2
from urllib2 import urlopen
from bs4 import BeautifulSoup
import csv
#imports and opens a .csv with containing all urls
urls = csv.reader(urlopen("https://www.pythonanywhere.com/user/mcca2174/files/home/mcca2174/Dropbox/pyanywhere/urllist.csv?", "rU"), dialect=csv.excel_tab)
#declare the .csv writer
file = open('/home/mcca2174/Dropbox/pyanywhere/Python_E/scrape_results.csv', 'wb')
writer = csv.writer(file)
#for loop which imports urls for data collection, establishes .csv writer, prints data to terminal and writes data to .csv file locally
for url in urls:
#pulls urls into the loop for scraping
site = urllib2.urlopen(url[0])
#massages webpage information for more accessible scraping
HIBSoup = BeautifulSoup(site, "html.parser")
#for loop which renders webpage text according to specific html tags
for content in HIBSoup.find_all('item'):
for title in content.find('title'):
A = title
for creator in content.find('dc:creator'):
#encoding to perserve German characters containing umlauts in the data set
B = creator.encode('utf-8')
for links in content.find('link'):
C = links
for comments in content.select('description'):
#ascii encoding to address non-ascii characters
D = comments.text.encode('ascii','replace')
for publication_date in content.find('pubdate'):
E = publication_date
for site in content:
#sort data for excel viewing
sort_data = [A, B, C, D, E]
Title = sort_data[0]
Author = sort_data[1]
Date = sort_data[4]
Link = sort_data[2]
Comment = sort_data[3]
sorted = [Title, Author, Date, Link, Comment]
writer.writerow(sorted)
print sorted
There, I got it. The directory issue is solved, but how I'm having an issue with my line that reads
site = urllib2.urlopen(url[0])
The error returned was
Traceback (most recent call last):
File "/home/mcca2174/opencsvtest.py", line 20, in <module>
site = urllib2.urlopen(url[0])
File "/usr/local/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/local/lib/python2.7/urllib2.py", line 392, in open
protocol = req.get_type()
File "/usr/local/lib/python2.7/urllib2.py", line 254, in get_type
raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: !DOCTYPE html
>>>
edit by admin to fix formatting
FYI:
In order to get code blocks to work correctly you need to precede them by a blank line.
So, this will work:
import random # Has a blank line directly before " :::python"
And this won't:
:::python
import random # Does NOT have a blank line directly before " :::python"
@mcca2174, it looks like you're trying to treat a line from an html file as a URL. "!DOCTYPE html" is not a url and so urllib is complaining. Check the response you get from your request for the list of urls.
Also, it may just be a formatting error, but your loop only encompasses the setting of site, so all the processing will only happen to the last url.
I will look into the html issue and get back, thanks!
Regarding the formatting, this is an error that occurred in through my own lack of experience posting code on this forum. In my code, there is the one large For loop and everything else below is contained within. For this I apologize because I realize how important formatting is when diagnosing. I would however like to thank those who have chimed in. The response to help has far exceeded my expectations. A bunch of dedicated individuals the python community are. :)
