UPDATE: Lots and lots of folks were good enough to volunteer – enough that it brought down the database for a few hours. We’re back in business, as of 10:30 pm – please come to the wiki and join us to do some data entry.
Like most people, I’ve been watching news coverage of the aftermath of Hurricane Katrina and feeling relatively helpless, wanting to do more than just donate to the Red Cross and other relief efforts. Yesterday afternoon, I received an email from David Geilhufe, who was helping to organize an interesting and important tech project: building a single database to resolve the dozens of “people missing” and “people safe” databases that have appeared on the web in the past week.
If you’re searching for a missing friend or loved one – like a friend who’s visiting this weekend and is looking for her grandfather – you currently need to check dozens of online databases. If this project succeeds, there will be a single database that resolves all the outstanding information.
I’ve been involved with a very small – but very labor intensive – part of this process – assigning sections of bulletin boards to volunteers to enter into the database. This has involved writing some quick scripts that group message board posts into sets of 25, generate web pages with links to them, and then create wiki pages so that folks can sign up for 25 entries to enter into the database.
We need two types of help on this project right now. If you’ve got a free hour to do data entry, we would love your help. Go to the Volunteer page of the wiki, follow instructions, choose and claim the chunk you’d like responsibility for. And please give me feedback on whether the process makes sense to you or whether the instructions are too confusing.
If you’ve got some wiki, Perl or PHP skillz and want to put in a bit more time, please let me know – I’m working through a list of unstructured data sources and writing sloppy hacks to generate chunks to assign to volunteers. I’d love help working through this, but this is an assignment that requires some ability with scripting and a bit of creativity. If you’re willing to give me a hand, I’d be very grateful – email me at ethanz AT gmail DOT com.
awesome idea, I posted the link around a few places and took on a block. It’s hard though as the entries are so free form…
Ethan: I got the good folks at About.com to compile the most complete list of lists of the missing I’ve seen yet:
http://www.about.com/relief.htm
Add to it this effort, which is coming up tonight:
http://www.thekatrinaportal.com/
Yahoo was talking about doing something.
You are absolutely right: The big need is to bring together this data so the connections that can be made are made.
Is there any way to scrape and at least make searchable to data in all these lists?
Jeff – this is a classic hard-to-automate problem. These posts are amazingly free form. Folks are automating the stuff that’s in structured databases – we’re working on the stuff that only succumbs to distributed human effort…
Thanks for the About link – will make sure we know about all those dbs…
Pingback: Branch, Leaf, and Idea » Katrina + Code, Data entry.
I have a similar idea — only faster. I’m building a distributable, caching screen-scraper! I only need 12 perl coders — no data entry necessary to solve this problem! Email me at katrina+vol@mixolydian.org to volunteer. (Be sure to mention which project you’re volunteering for!)
Obviously you’re welcome to do whatever you like, Vanessa, but there are a lot of developers working on this project. You might want to contact David Geilhufe and try to coordinate your efforts with him.
Ethan,
I’m going to suggest that a better approach might be an indexing or searching system, rather than actual aggregation, given the complexity of the free-form data.
That said, I did some work on scraping this sort of stuff last year and will take a shot at one of those sites, if you’ll point me at it. (I suspect it’ll have to be customized for almost every site.)
Adam –
I think we may be doing something closer to indexing than you think. I’m simply assigning chunks of bulletin boards to people. People are then reading those bulletin boards and creating records in a database. That was a standalone SQL database – now it’s something run by Salesforce.com, probably a beast of an Oracle db. So we’re not really aggregating, except to assign people chunks of existing systems to tag and code into a database…
Ethan,
Here is simple way to help organize who is where and help projects like Katrina people finder:
1) News crews video tape people speaking their name, where they are from, and where they currently are (and date). Perhaps additional information.
2) Video sent to closed captioning service to caption (automated/manuel)
3) Google indexes video via closed captioning.
Voila – images & video and information on everyone displaced from storm.
i need more info
I Want to be part of this will be happy for some more info.
very nice working ur doing
Hey I would like to be part of this
Comments are closed.